openpecha/Finetuned_Alibaba_Large

SentenceTransformer based on Alibaba-NLP/gte-large-en-v1.5

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-large-en-v1.5. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Alibaba-NLP/gte-large-en-v1.5
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Who gave the Dalai Lama a lecture before he left Lhasa, and what was it about?',
    "Before the Dalai Lama left Lhasa, he was given a long lecture by General Chang Chin-wu, the permanent representative of China. The lecture covered several topics, including recent events in Hungary and Poland, the solidarity of socialist powers, the Dalai Lama's visit to India, and specific instructions on how to handle questions about the Indo-Tibetan frontier and the situation in Tibet. General Chang Chin-wu also suggested that the Dalai Lama prepare his speeches in advance.",
    'Everywhere I went, I was accompanied by a retinue of servants. I was surrounded by government ministers and advisors clad in sumptuous silk robes, men drawn from the most exalted and aristocratic families in the land.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: all-nli-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9924
dot_accuracy	0.0076
manhattan_accuracy	0.9924
euclidean_accuracy	0.9898
max_accuracy	0.9924

Training Details

Training Dataset

Unnamed Dataset

• Size: 7,075 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 17.9 tokens max: 33 tokens	min: 10 tokens mean: 96.59 tokens max: 810 tokens	min: 8 tokens mean: 90.43 tokens max: 810 tokens

• Samples:

  anchor	positive	negative
  What was the Dalai Lama's plan for the senior members of the Government if the situation worsened?	Shortly afterwards, with the Chinese consolidating their forces in the east, we decided that I should move to southern Tibet with the most senior members of Government. That way, if the situation deteriorated, I could easily seek exile across the border with India. Meanwhile, Lobsang Tashi and Lukhangwa were to remain in Lhasa in an acting capacity: I would take the seals of state with me.	The Dalai Lama's press conference on 20 June had a significant impact on the international perception of the Tibetan issue. By formally repudiating the Seventeen-Point Agreement and detailing the atrocities committed against Tibetans, the Dalai Lama aimed to present a truthful account of the situation in Tibet. This press conference received wide coverage and helped to counter the Chinese government's narrative. However, despite the extensive media attention, the Dalai Lama acknowledged the challenges in overcoming the Chinese government's efficient public relations campaign and the general reluctance of the international community to face the truth about the situation in Tibet. The press conference marked an important step in raising global awareness about the Tibetan struggle and the injustices faced by its people.
  What did the young Dalai Lama enjoy about the opera festival?	They gave their performances on a paved area situated on the far side of, but adjacent to, the Yellow Wall. I myself watched the proceedings from a makeshift enclosure erected on the top of one of the buildings that abutted the wall on the inside.	This man had become notorious in Lhasa because of his close association with the Chinese occupation forces. Earlier that morning he had attended a daily congregation of monastic officials called the Trungcha Ceremony, and for some unknown reason, about eleven o'clock, he rode towards the Norbulingka on a bicycle, wearing a semi-Chinese dress, dark glasses and a motorcyclist's dust mask, and carrying a pistol unconcealed in his belt. Some of the crowd took him for a Chinese in disguise; others thought he was bringing a message from the Chinese headquarters. Their anger and resentment against everything Chinese suddenly burst into fury, and murder was the tragic result.
  What is the Tibetan term "Lama" equivalent to in Indian terminology?	Actually, Dalai is a Mongolian word meaning 'ocean' and Lama is a Tibetan term corresponding to the Indian word guru, which denotes a teacher.	The Chinese authorities handled the issue of Tibetan language and culture with a systematic and ruthless approach aimed at eradicating Tibetan identity. They implemented policies that severely suppressed Tibetan culture and language. For instance, the education provided to Tibetans was primarily conducted in Chinese, with a stated goal of eradicating the Tibetan language within fifteen years. Many schools were essentially labor camps for children, and only a select few Tibetan students received proper education, which was conducted in China to foster 'unity'. Additionally, the Chinese authorities brutally suppressed Tibetan culture by banning formal religion, desecrating thousands of monasteries and nunneries, and enforcing policies that controlled the Tibetan population through measures such as forced abortions and sterilizations. The Chinese also exploited Tibet's natural resources and transformed its economy in ways that primarily benefited China, leaving Tibetans in a state of abject poverty and environmental degradation.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 393 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 18.13 tokens max: 30 tokens	min: 10 tokens mean: 99.75 tokens max: 810 tokens	min: 10 tokens mean: 99.99 tokens max: 810 tokens

• Samples:

  anchor	positive	negative
  What was the role of the Dalai Lama in the feudal system of Tibet?	The Dalai Lama held a unique and central role in the feudal system of Tibet, combining both lay and monastic authority. He had two prime ministers, one a monk and one a layman, and most other offices were duplicated to reflect this dual nature. The Dalai Lama was the ultimate source of justice and was regarded with the highest reverence by the people, who saw him as the incarnation of Chenresi. This reverence ensured that the Dalai Lama could not become an unjust tyrant, providing a final appeal to a source of justice that the people could absolutely trust.	The Dalai Lama and his companions faced numerous challenges while crossing the high mountains. They had to traverse slippery and muddy tracks, often leading to heights of over 19,000 feet where snow and ice were still present. The journey involved crossing particularly high and steep passes, such as the Yarto Tag-la, where some ponies could not climb the track, necessitating dismounting and leading them. They endured long hours of hard riding and climbing, often becoming very tired and saddle-sore. The weather posed significant difficulties, including snowstorms, snow glare, torrential rain, and strong winds that picked up snow and whirled it into their faces. The cold was intense, numbing their fingers and hands, and causing ice to form on their eyebrows and moustaches. Additionally, they had to deal with the threat of being spotted by Chinese aircraft, which added to their unease and forced them to divide into smaller parties. The journey was further complicated by a duststorm and the glare from the snow, which was particularly hard on those without goggles. Finally, the weather did its worst when they reached Mangmang, where they experienced heavy rain that leaked into their tents, causing discomfort and illness.
  What was the Dalai Lama's impression of Prime Minister Shastri?	The Dalai Lama held Prime Minister Lal Bahadur Shastri in high regard, respecting him greatly. He appreciated Shastri's friendship and political support for the Tibetan refugees, noting that Shastri was even more of a political ally than Nehru. The Dalai Lama admired Shastri's powerful mind and spirit, describing him as a bold and decisive leader despite his frail appearance. Shastri's compassion and strict vegetarianism, stemming from a childhood incident, also left a lasting impression on the Dalai Lama. The Dalai Lama mourned Shastri's death deeply, recognizing the loss of a true and mighty friend, an enlightened leader, and a genuinely compassionate spirit.	The Dalai Lama's initial impression of the Chinese general's appearance was that he looked extremely drab and insignificant among the splendid figures of his own officials. The Dalai Lama observed the general and his aides in gray suits and peaked caps, which contrasted sharply with the red and golden robes of the Tibetan officials. This drabness, as the Dalai Lama later reflected, was indicative of the state to which China would reduce Tibet. However, the general turned out to be friendly and informal during their meeting.
  What were the names of the two Lhasa Apso dogs?	The names of the two Lhasa Apso dogs were Sangye and Tashi.	The Dalai Lama's journey was marked by challenging weather conditions. During the journey, they faced an 'extraordinary sequence of snowstorms, snow glare, and torrential rain.' At one point, while crossing the Lagoe-la pass, they encountered a 'heavy storm' which made it 'very cold,' numbing their fingers and hands, and freezing their eyebrows. Additionally, they experienced a duststorm and intense snow glare. The weather did its worst when they reached Mangmang, where it 'began to pour with rain,' causing leaks in the tents and resulting in a sleepless night for many, including the Dalai Lama, who felt very ill the next morning.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	all-nli-dev_max_accuracy
0	0	-	-	0.8830
0.0565	50	0.7484	0.2587	0.9873
0.1130	100	0.2822	0.2313	0.9898
0.1695	150	0.3023	0.2291	0.9873
0.2260	200	0.2484	0.2155	0.9873
0.2825	250	0.2909	0.1965	0.9847
0.3390	300	0.2999	0.2008	0.9847
0.3955	350	0.2586	0.1670	0.9924
0.4520	400	0.2385	0.1467	0.9898
0.5085	450	0.2353	0.1311	0.9898
0.5650	500	0.2632	0.1340	0.9873
0.6215	550	0.3793	0.1218	0.9898
0.6780	600	0.1978	0.1174	0.9898
0.7345	650	0.179	0.1254	0.9898
0.7910	700	0.1326	0.1142	0.9924
0.8475	750	0.1842	0.1153	0.9924

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.2.1
• Accelerate: 0.31.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}