deberta-v3-base-formality

microsoft/deberta-v3-base fine-tuned for binary English text formality classification.

The model outputs one of two labels:

id	label
0	INFORMAL
1	FORMAL

Model details

Base model	microsoft/deberta-v3-base
Architecture	DebertaV2ForSequenceClassification (single linear classification head, 2 logits)
Parameters	184.4M
Tokenizer	microsoft/deberta-v3-base (SentencePiece, vocab 128100)
Max sequence length	128 tokens
Language	English
Precision used in training	bf16

Training data

Fine-tuned on osyvokon/pavlick-formality-scores, the crowdsourced formality annotations of Pavlick & Tetreault (2016). Each sentence carries a continuous avg_score on a 7-point Likert scale from -3 (very informal) to +3 (very formal); sentences come from four domains (answers, blog, email, news).

Label derivation

The original annotation is continuous; the binary labels used here are derived, not provided by the annotators:

label = FORMAL    if avg_score >  0.0
label = INFORMAL  if avg_score <= 0.0

The threshold is the documented neutral midpoint of the annotation scale (0.0). It was fixed a priori from the scale definition and was never tuned on the test split.

Data preparation

Step	Detail
Missing text / score	dropped
Empty / whitespace-only text	dropped
Exact duplicate texts	dropped from the train pool (official test split kept intact)
Train/test leakage	train rows whose text also appears in the test split were removed
Text normalisation	none - the raw sentence is passed to the tokenizer
Splits	official train split (minus a stratified 10% validation carve-out, seed 42) for training; official test split held out

Split	Examples	INFORMAL	FORMAL
train	8318	4187	4131
validation	925	466	459
test	2000	983	1017

Classes are close to balanced, so the model is trained with standard (unweighted) cross-entropy.

Training hyperparameters

Optimizer	AdamW
Learning rate	2e-05
LR schedule	linear, warmup ratio 0.1 (156 steps)
Epochs	3.0
Train batch size	16 (effective 16)
Eval batch size	32
Weight decay	0.01
Max grad norm	1.0
Padding	dynamic (DataCollatorWithPadding)
Seed	42
Model selection	best validation macro-F1, evaluated once per epoch
Hardware	NVIDIA GeForce RTX 5060 Ti (15.9 GB)
Training time	3m 22.5s

Evaluation

Evaluated once on the held-out official test split (2000 sentences). The test split was not used for threshold selection, hyperparameter tuning, early stopping or model selection.

Metric	Value
Accuracy	0.8190
Macro Precision	0.8220
Macro Recall	0.8181
Macro F1	0.8183
ROC-AUC	0.9014
MCC	0.6402

Per class:

Label	Precision	Recall	F1	Support
INFORMAL	0.8493	0.7681	0.8066	983
FORMAL	0.7948	0.8682	0.8299	1017

Confusion matrix (rows = true, columns = predicted):

	pred INFORMAL	pred FORMAL
true INFORMAL	755	228
true FORMAL	134	883

Agreement with the original continuous annotation on the test split (avg_score vs. predicted P(FORMAL)): Pearson r = 0.7794, Spearman rho = 0.7959.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "thealper2/deberta-v3-base-formality"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

text = "Could you please provide the requested document?"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]

print(model.config.id2label[int(probs.argmax())], float(probs.max()))
# -> FORMAL 0.9...

With pipeline:

from transformers import pipeline

clf = pipeline("text-classification", model="thealper2/deberta-v3-base-formality", top_k=None)
clf("hey u wanna grab some food later")

Limitations

• Derived labels. The source annotation is a continuous score; binarising it at the scale midpoint discards the intensity of formality and makes sentences near the threshold intrinsically ambiguous. Model probabilities near 0.5 should be treated as "unclear", not as a confident decision.
• Ties. Sentences with avg_score exactly at the threshold are mapped to INFORMAL by convention.
• Domain coverage. Training data covers Yahoo! Answers, blogs, email and news sentences. Behaviour on other domains (chat logs, code, transcripts, non-native or dialectal English) is untested.
• Sentence-level. Trained on single sentences of about 20.9 tokens on average; long multi-paragraph inputs are truncated at 128 tokens.
• English only.
• Annotation subjectivity. Formality judgements are subjective and the gold scores are crowd averages; the ceiling of this task is bounded by annotator agreement.

Citation

Training data:

@article{pavlick2016empirical,
  title   = {An Empirical Analysis of Formality in Online Communication},
  author  = {Pavlick, Ellie and Tetreault, Joel},
  journal = {Transactions of the Association for Computational Linguistics},
  volume  = {4},
  pages   = {61--74},
  year    = {2016}
}

Base model:

@inproceedings{he2023debertav3,
  title     = {DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing},
  author    = {He, Pengcheng and Gao, Jianfeng and Chen, Weizhu},
  booktitle = {ICLR},
  year      = {2023}
}