---
library_name: peft
license: apache-2.0
datasets:
- maastrichtlawtech/lleqa
language:
- fr
metrics:
- rouge
- meteor
pipeline_tag: text-generation
inference: false
tags:
- legal
---

# wizardLM-7b-v1.0-lleqa

This is a [wizardlm-7b-v1.0](https://huggingface.co/WizardLM/WizardLM-7B-V1.0) model fine-tuned with [QLoRA](https://github.com/artidoro/qlora) for long-form legal question answering in **French**. 

## Usage

```python
[...]
```

## Training

#### Data

We use the [Long-form Legal Question Answering (LLeQA)](https://huggingface.co/datasets/maastrichtlawtech/lleqa) dataset to fine-tune the model. LLeQA is a French native dataset for studying legal information retrieval and question answering. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.

#### Hyperparameters

We fine-tune the model through 4-bit QLoRA finetuning with an effective batch size of 8 for 10 epochs (i.e., 1.1K steps) using paged AdamW optimizer with default momentum parameters and constant learning rate schedule of 2e-4. We employ NormalFloat4 with double quantization for the base models and add LoRA adapters on all linear layers by setting r=16, alpha=32 while utilizing float16 as computation datatype. Additionally, we perform NTK-aware scaling of RoPE to extend the context window to 4096 tokens. Training takes around 7.5 hours to complete on a single Tesla V100 GPU with 32GBs of memory. More details can be found in this [paper](https://arxiv.org/abs/) and [repository](https://github.com/maastrichtlawtech/lleqa).

## Citation

```bibtex
@article{louis2023interpretable,
  author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
  title = {Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models},
  journal = {CoRR},
  volume = {abs/2309.17050},
  year = {2023},
  url = {https://arxiv.org/abs/2309.17050},
  eprinttype = {arXiv},
  eprint = {2309.17050},
}
```