`NLLB-LLM2Vec': Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages

Repository: https://github.com/fdschmidt93/trident-nllb-llm2vec
Paper: https://arxiv.org/abs/2406.12739

NLLB-LLM2Vec multilingually extends LLM2Vec via efficient self-supervised distillation. We train the up-projection and LoRA adapters of the NLLB-LLM2Vec by forcing its mean-pooled token embeddings to match (via mean-squared error) the output of the original LLM2Vec.

This model has only been trained on self-supervised data and not yet been fine-tuned on any downstream task! This version is expected to perform better than self-supervised adaptation in the original paper, as LoRAs are merged into the model prior to task fine-tuning. The backbone of this model is LLM2Vec-Meta-Llama-31-8B-Instruct-mntp-unsup-simcse. We use the encoder of NLLB-600M.

⚠️ Make sure that you correctly set the src_lang (i.e., AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", src_lang=LANG_CODE) for the language you are using NLLB-LLM2Vec with! You can find a list of supported languages here

Usage

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("fdschmidt93/NLLB-LLM2Vec-Meta-Llama-31-8B-Instruct-mntp-unsup-simcse")
model = AutoModel.from_pretrained(
    "fdschmidt93/NLLB-LLM2Vec-Meta-Llama-31-8B-Instruct-mntp-unsup-simcse",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda" if torch.cuda.is_available() else "cpu",
)

flores_en = [
    "Lead researchers say this may bring early detection of cancer, tuberculosis, HIV and malaria to patients in low-income countries, where the survival rates for illnesses such as breast cancer can be half those of richer countries.",
    "Enceladus is the most reflective object in the solar system, reflecting about 90 percent of the sunlight that hits it.",
]

en_embeds = model.encode(flores_en)

flores_de = [
    "Führende Forscher sagen, dass dies die Früherkennung von Krebs, Tuberkulose, HIV und Malaria für Patienten in einkommensschwachen Ländern fördern könnte, wo die Überlebensraten bei Krankheiten wie Brustkrebs teilweise nur halb so hoch sind wie in reicheren Ländern.",
    "Enceladus ist das Objekt im Sonnensystem, das am stärksten reflektiert. Er wirft etwa 90 Prozent des auf ihn treffenden Sonnenlichts zurück.",
]

de_embeds = model.encode(flores_de, src_lang="deu_Latn")


# Compute cosine similarity
en_embeds_norm = F.normalize(en_embeds, p=2, dim=1)
de_embeds_norm = F.normalize(de_embeds, p=2, dim=1)
cos_sim = en_embeds_norm @ de_embeds_norm.T

print(cos_sim)
"""
tensor([[0.9062, 0.0894],
        [0.1289, 0.8633]])
"""

Fine-tuning

You should fine-tune the model on labelled data unless you are using the model for unsupervised retrieval-style tasks. NLLB-LLM2Vec supports both AutoModelForSequenceClassification and AutoModelForTokenClassification.

import torch
from transformers import AutoModelForSequenceClassification
from peft import get_peft_model
from peft.tuners.lora.config import LoraConfig

# Only attach LoRAs to the linear layers of LLM2Vec inside NLLB-LLM2Vec
lora_config = LoraConfig(
    rank = 16,
    lora_alpha = 32,
    target_modules = r".*llm2vec.*(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj).*",
    bias = "none",
    task_type = "SEQ_CLS"
  )
model = AutoModelForSequenceClassification.from_pretrained(
    "fdschmidt93/NLLB-LLM2Vec-Meta-Llama-31-8B-Instruct-mntp-unsup-simcse",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)
model = get_peft_model(model, lora_config)

Questions

If you have any question about the code, feel free to email Fabian David Schmidt (fabian.schmidt@uni-wuerzburg.de).

Citation

If you are using NLLB-LLM2Vec in your work, please cite

@misc{schmidt2024selfdistillationmodelstackingunlocks,
      title={Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages}, 
      author={Fabian David Schmidt and Philipp Borchert and Ivan Vulić and Goran Glavaš},
      year={2024},
      eprint={2406.12739},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.12739}, 
}

The work has been accepted to findings of EMNLP. The Bibtex will therefore be updated when the paper will be released on ACLAnthology.