vaschko/xlm-roberta-base-unfair-clauses

XLM-RoBERTa base, fine-tuned to detect unfair clauses in Terms of Service

A sentence classifier that answers one question: is this sentence a potentially unfair clause? It returns a probability, nothing else. Given a German or English sentence from consumer terms of service, it flags the kinds of clauses the CLAUDETTE project annotates as unfair: unilateral changes and termination, limitation of liability, jurisdiction and choice-of-law clauses, arbitration, content removal, contract by use, and privacy-related clauses.

The model was built for a bachelor's thesis (2026) as the selection stage of a browser extension that summarizes terms and conditions and highlights red flags. In that pipeline the classifier picks the sentences worth reporting and ranks them; a local LLM then assigns a category and writes the explanation. Code, evaluation harness and training script: https://github.com/yetiiil/SummarizationBenchmark (Testing/Scripts/Evaluation/RedFlags/).

Intended use

• Pre-filtering and ranking of sentences in consumer terms of service, so that a reader (or a downstream model) looks at the few sentences that matter.
• Research on unfair-clause detection, as a fine-tuned baseline on the CLAUDETTE corpus.

Not intended as legal advice, and not licensed for commercial use (see License).

How it is used in the extension

Each section of a document is split into sentences (with an abbreviation-aware splitter for legal German and English). All sentences are scored; those with a probability of at least 0.85 are kept, sorted by probability, and capped at five per section. The kept sentences are quoted verbatim, so every flag is grounded in the text by construction.

Training data

CLAUDETTE multilingual corpus (Drawzeski et al., 2021): terms of service of 25 online platforms, sentence-level annotations for nine unfair clause types and four fairness levels. Only the German and English versions were used. The split follows the corpus and is by company, so no company appears in more than one split.

Split	Companies	Documents	Sentences	Unfair sentences
train	20	40	9,858	943 (9.6 %)
validation	2	4	819	88
test	3 (Weebly, Yelp, Zynga)	6	2,065	188

Positive label: the sentence is annotated potentially_unfair or clearly_unfair. Everything else (untagged, clearly_fair) is negative.

Training procedure

• Base model: FacebookAI/xlm-roberta-base, with a single-logit classification head (num_labels=1, sigmoid at inference).
• Loss: BCEWithLogitsLoss with pos_weight = negatives / positives (about 9.5), because only 9.6 % of the sentences are positive.
• AdamW, learning rate 2e-5, batch size 8, maximum length 192 tokens (covers over 99 % of the corpus sentences), 10 epochs, seed 20260810, trained on an Apple GPU (MPS).
• After every epoch the model was scored on the validation split and the decision threshold tuned for F1. The released checkpoint is epoch 9 with threshold 0.85 (validation P 0.750 / R 0.716 / F1 0.733). threshold.json and history.json in this repository record that choice and the full per-epoch history.

cd Testing/Scripts/Evaluation/RedFlags
PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 python3 classifier_train.py --epochs 10 --batch-size 8 --max-length 192

Training on Apple GPUs is not bit-exact; a retrain gives comparable, not identical numbers (the spread between epochs was about ±0.04 F1).

Evaluation

All figures at threshold 0.85. The CLAUDETTE numbers were measured the way the extension uses the model: whole documents are reconstructed, split by the application's own sentence splitter, scored, and the flagged spans are mapped back to the annotated sentences.

CLAUDETTE test split (DE + EN, 188 annotated unfair sentences)

	Precision	Recall	F1
Exact sentence	0.705	0.750	0.727
Within one sentence of the annotation	0.806	0.883	0.843

Precision@5 over the six test documents: 0.867; precision@10: 0.833.

Recall per annotated clause type:

Clause type	Gold sentences	Recall
Arbitration	10	0.30
Unilateral change	26	0.73
Content removal	22	0.86
Jurisdiction	14	0.93
Choice of law	10	0.70
Limitation of liability	70	0.77
Unilateral termination	40	0.90
Contract by using	16	0.63
Privacy included	4	0.50

External check: LexGLUE UNFAIR-ToS (coastalcph/lex_glue, config unfair_tos, English). This dataset descends from the same project, and 1,779 of its sentences occur verbatim in the training or validation data. Those were removed first; on the remaining 7,351 sentences (834 positive) the model reaches P 0.758 / R 0.729 / F1 0.743. On the overlapping sentences the two corpora agree on the label in 97.4 % of cases, so the label definitions are compatible.

In the shipped pipeline (classifier selects, gemma4:e4b labels and explains), the same test split gives P 0.737 / R 0.702 / F1 0.719. The prompted LLM alone, without the classifier, reached F1 0.305.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "vaschko/xlm-roberta-base-unfair-clauses"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

sentences = [
    "We may terminate your account at any time, for any reason, without notice.",
    "Diese Bedingungen gelten für alle Bestellungen über unseren Online-Shop.",
]
batch = tokenizer(sentences, truncation=True, max_length=192, padding=True, return_tensors="pt")
with torch.no_grad():
    probabilities = torch.sigmoid(model(**batch).logits.squeeze(-1))

for sentence, p in zip(sentences, probabilities.tolist()):
    print(f"{p:.2f}  {'unfair' if p >= 0.85 else 'ok'}  {sentence}")

The threshold of 0.85 is the one tuned on the validation split; lower it to trade precision for recall.

Limitations

• Binary only. The model does not say which kind of clause a sentence is; in the extension that is left to the LLM.
• Trained and evaluated on German and English. The base model is multilingual, but nothing was measured for other languages.
• Sentence-level. A clause that spans several sentences produces one score per sentence.
• The corpus consists of terms of service of online platforms, annotated between 2019 and 2021. Other contract types (leases, insurance, employment) are out of domain.
• Arbitration clauses are the weakest category on the test split (recall 0.30 on 10 sentences); the support is too small for a firm conclusion.
• The two evaluation corpora are related; the external figure is therefore a check against memorisation, not a fully independent benchmark.
• Output is a statistical estimate, not a legal assessment.

License

The weights are released under CC BY-NC 4.0: attribution required, non-commercial use only.

The non-commercial restriction is inherited from the training data. The CLAUDETTE multilingual corpus is licensed CC BY-NC 2.5; whether fine-tuned weights are a derivative work of their training data is legally unsettled, and this release takes the conservative reading and passes the restriction on. The base model xlm-roberta-base is MIT licensed.

References

• Lippi, M., Pałka, P., Contissa, G., Lagioia, F., Micklitz, H.-W., Sartor, G., Torroni, P. (2019). CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27, 117–139.
• Drawzeski, K., Galassi, A., Jabłonowska, A., Lagioia, F., Lippi, M., Micklitz, H.-W., Sartor, G., Tagiuri, G., Torroni, P. (2021). A Corpus for Multilingual Analysis of Online Terms of Service. Proceedings of the Natural Legal Language Processing Workshop 2021.
• Galassi, A., Lagioia, F., Jabłonowska, A., Lippi, M. (2024). Unfair clause detection in terms of service across multiple languages. Artificial Intelligence and Law.
• Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D. M., Aletras, N. (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. ACL 2022.
• Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020.