AMANA-QE

A research model from AMANA, a study of clinical meaning preservation in English → Arabic medication instructions. Given an English instruction and an Arabic translation, it returns a score for how likely the translation is to keep the instruction's meaning.

It is a ranking score, not a safety check. It must not be used to approve translations for patients, and it is not a medical device. It is published so the results in the repository can be reproduced and compared against.

How to use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("momen-ibrahim/amana-qe")
model = AutoModelForSequenceClassification.from_pretrained("momen-ibrahim/amana-qe").eval()

src = "Take 1 tablet of levothyroxine 25 mcg by mouth once daily."
hyps = ["خذ قرصاً واحداً من ليفوثيروكسين 25 ميكروغرام عن طريق الفم مرة واحدة يومياً.",
        "خذ قرصاً واحداً من ليفوثيروكسين 25 ملغ عن طريق الفم مرة واحدة يومياً."]  # mcg written as mg
enc = tok([src] * len(hyps), hyps, truncation=True, max_length=192, padding=True, return_tensors="pt")
with torch.no_grad():
    score = torch.softmax(model(**enc).logits, -1)[:, 1]  # label 1 = meaning kept
print(score)

Compare scores between candidate translations of the same source. Do not read a score as a probability or apply a fixed threshold: the evidence below shows that fails.

Training

LaBSE (sentence-transformers/LaBSE) fine-tuned as a two-class cross-encoder for 2 epochs (learning rate 2e-5, batch 32, seed 13). The training data is the train split of AMANA-Bench: Arabic translations of generated English medication instructions, each with correct versions and versions with one critical error (dose, strength, unit, frequency, duration, timing, route, laterality, negation, omission or drug). The split is by drug, so the 13 test drugs were never seen in training. The generator, the labels and this model were all made by one author.

Evaluation

track	AUROC	false alarms at 0.5	expected calibration error
AMANA-Bench, held-out drugs (same templates)	0.999	0%	0.009
TICO-19 real text with one injected number or negation error	0.80	70%	0.46
Error types left out of training: unit, laterality, negation (5 seeds; recall at 0.5 is 7–36%)	0.66–0.73	0–6%	0.31–0.46
Dosage forms left out of training: eye drops, inhalers (5 seeds)	0.85–0.86	70–100%	0.26–0.33

The last two rows retrain the model without those error types or forms, so they are separate models, not this one.

Read the 0.999 against a source-blind baseline. A character n-gram classifier that never sees the English reaches AUROC 0.711 on the same held-out drugs, and 0.94 for unit, route and negation errors, because correct instructions in the benchmark never use some words that errors use. Much of this model's benchmark score can come from the Arabic alone. The drop to 0.66–0.73 (mean 0.69) on unseen error types is the clearest sign that it learned the generator.

Limitations

• Trained only on generated instructions from one template grammar, in Modern Standard Arabic.
• Not calibrated: at 0.5, false alarms range from 0% to 100% across tracks and seeds.
• Not evaluated by any clinician, and not tested on naturally occurring translation errors.
• The headline model is one seed.

Licence and citation

The weights are released under Apache 2.0, the licence of the LaBSE base model. AMANA-Bench is CC BY 4.0. Please cite the repository (its CITATION.cff).