LLaRA: LLaMA Relevance Assessor

LLaRA (LLaMA Relevance Assessor) is a fine-tuned open-source large language model based on LLaMA 3.1 8B, specialized in document relevance assessment. It has been trained using data from the MS MARCO Deep Learning Track 2022, focusing on learning to evaluate the relevance of query-document pairs as performed in information retrieval tasks.

---

🔍 Model Overview

• Model Name: LLaRA
• Base Model: Meta LLaMA 3.1 8B
• Task: Relevance Assessment for Information Retrieval
• Training Data: MS MARCO Deep Learning Track 2022 (queries, documents, qrels)
• License: Apache 2.0
• Context Length: 128k tokens (extended context support)
• Output Format: JSON-style relevance scores (e.g., {"M": x, "T": y, "O": z})

---

🚀 Use Cases

LLaRA is designed for scenarios that require high-quality automated relevance assessments, such as:

• Evaluating search system output against known judgments
• Ranking or re-ranking documents in response to a query
• Simulating expert annotator judgments in IR pipelines
• Large-scale IR evaluation where human labeling is costly or infeasible

---

🛠️ Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("your-username/llara")
model = AutoModelForCausalLM.from_pretrained("your-username/llara")

prompt = '''[QUERY]
What are the symptoms of COVID-19?

[DOCUMENT]
COVID-19 typically causes fever, cough, fatigue, and in some cases loss of smell. It can lead to severe respiratory issues in vulnerable individuals.

[INSTRUCTION]
Assess the relevance of the document to the query on a 0-2 scale.
Return: {"M": ..., "T": ..., "O": ...}
'''

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📊 Training Details

• Training Framework: PyTorch / Transformers
• Compute: Multi-GPU setup (NVIDIA RTX 2080 Ti, etc.)
• Methodology: Instruction-style fine-tuning using stratified and pooled sampling strategies
• Prompt Format: Task-oriented with embedded query and document context
• Evaluation: Compared against TREC assessor scores and other LLM baselines

---

📄 License

This model is licensed under the Apache License 2.0. You are free to use, modify, and distribute this model with proper attribution.

---

🤝 Acknowledgments

This project builds upon:

• Meta AI’s LLaMA 3.1
• MS MARCO Dataset
• TREC Deep Learning Track

---

✍️ Citation

If you use LLaRA in your work, please consider citing or acknowledging the project in your publications and projects.

---

📫 Contact

For questions, collaborations, or contributions, feel free to open an issue or reach out here