kevin-rice/embeddinggemma-ticket-similarity

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kevin-rice/embeddinggemma-ticket-similarity")
# Run inference
queries = [
    'Misaligned Template Section Fields and Inconsistent Invoice Layout Compared to UPS Orders',
]
documents = [
    'PDF export button icon appears similar to Excel icon in Inventory Master List report',
    'Inventory Master List Displays Active/Inactive Products While Manage Products Uses Different Status Visibility Logic',
    'SKU Toggle Prints Commodity Code (CC) Instead of SKU in Location Labels',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.3035, 0.6568, 0.2639]])

Evaluation

Metrics

Semantic Similarity

• Dataset: ticket-similarity-eval
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.8735
spearman_cosine	0.8192

Training Details

Training Dataset

Unnamed Dataset

• Size: 315 training samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 315 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 10 tokens mean: 16.62 tokens max: 27 tokens	min: 10 tokens mean: 16.83 tokens max: 27 tokens	min: 0.0 mean: 0.57 max: 1.0

• Samples:

  sentence1	sentence2	score
  View button icon under Action column is not displayed properly	Unable to Search and Select Product in Select Product Catalog During In-House Replenishment	1.0
  Pick Assignment Throws Replenishment Error Even When Primary Location Has Available Stock	Cycle Count Variance report not fetching latest cycle count data dynamically	0.8
  Move Items UI should auto-hide location selection when only one Primary location exists	Primary Location not populated when product is fetched using Scan/Search Barcode in In-House Replenishment	0.8

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss",
      "cos_score_transformation": "torch.nn.modules.linear.Identity"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 79 evaluation samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 79 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 10 tokens mean: 16.51 tokens max: 25 tokens	min: 11 tokens mean: 17.03 tokens max: 26 tokens	min: 0.0 mean: 0.63 max: 1.0

• Samples:

  sentence1	sentence2	score
  Update Packing Slip date format to MM-DD-YYYY	Accounting Template data is not fetching under Template column in Sales History By Item report	0.8
  Update Comments Section Format and Merge Herman ID / Employee ID Field	Order With Quantity Exceeding Available Primary Stock Is Marked Delivered Instead of Back Order and Creates Negative Stock	0.0
  Default distribution center comment is not displayed in Comments section	Update Packing Slip date format to MM-DD-YYYY	0.8

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss",
      "cos_score_transformation": "torch.nn.modules.linear.Identity"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• learning_rate: 2e-05
• warmup_steps: 0.1
• fp16: True
• per_device_eval_batch_size: 4

All Hyperparameters

Click to expand

• per_device_train_batch_size: 4
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 4
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	ticket-similarity-eval_spearman_cosine
0.0633	5	0.1581	-	-
0.1266	10	0.1581	-	-
0.1899	15	0.1117	-	-
0.2532	20	0.0869	0.0750	0.6907
0.3165	25	0.0651	-	-
0.3797	30	0.0590	-	-
0.4430	35	0.0580	-	-
0.5063	40	0.0698	0.1141	0.5602
0.5696	45	0.1079	-	-
0.6329	50	0.0932	-	-
0.6962	55	0.0762	-	-
0.7595	60	0.0938	0.0637	0.7089
0.8228	65	0.1259	-	-
0.8861	70	0.0735	-	-
0.9494	75	0.0276	-	-
1.0127	80	0.0551	0.0607	0.7692
1.0759	85	0.0788	-	-
1.1392	90	0.0807	-	-
1.2025	95	0.0334	-	-
1.2658	100	0.0508	0.0687	0.7471
1.3291	105	0.0719	-	-
1.3924	110	0.0404	-	-
1.4557	115	0.0143	-	-
1.5190	120	0.0740	0.0630	0.7372
1.5823	125	0.0410	-	-
1.6456	130	0.0483	-	-
1.7089	135	0.0629	-	-
1.7722	140	0.0513	0.0483	0.7610
1.8354	145	0.0175	-	-
1.8987	150	0.0397	-	-
1.9620	155	0.0341	-	-
2.0253	160	0.0223	0.0478	0.7755
2.0886	165	0.0167	-	-
2.1519	170	0.0230	-	-
2.2152	175	0.0600	-	-
2.2785	180	0.0357	0.0412	0.8031
2.3418	185	0.0479	-	-
2.4051	190	0.0172	-	-
2.4684	195	0.0183	-	-
2.5316	200	0.0213	0.0399	0.8162
2.5949	205	0.0115	-	-
2.6582	210	0.0305	-	-
2.7215	215	0.0101	-	-
2.7848	220	0.0189	0.0388	0.8229
2.8481	225	0.0249	-	-
2.9114	230	0.0104	-	-
2.9747	235	0.0099	-	-
3.0	237	-	0.0391	0.8192

Training Time

• Training: 46.3 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.7.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}