frankyy03/laya-pt-es-nli

Laya PT/ES NLI

This checkpoint fine-tunes convaiinnovations/laya-multilingual for three-way natural language inference in Portuguese and Spanish. It predicts contradiction, entailment, or neutral in one non-autoregressive forward pass.

The checkpoint keeps the original 322M-parameter Laya architecture. It adds no inference module. It also preserves exact input_ids and inputs_embeds parity, which is required by the separate audio-adapter experiment.

Use

pip install laya

import laya

agent = laya.load("frankyy03/laya-pt-es-nli")
questions = {
    "relation": {
        "type": "choice",
        "instructions": "Qual é a relação lógica entre o estado e a hipótese: Um animal está dormindo.",
        "criteria": {
            "contradiction": "o estado torna a hipótese falsa",
            "entailment": "o estado confirma a hipótese",
            "neutral": "o estado não confirma nem contradiz a hipótese",
        },
    }
}
result = agent.predict({"state": "Um gato dorme no sofá."}, questions)
print(result["answers"]["relation"])

Use localized instructions and option descriptions. The training and evaluation label order is contradiction, entailment, neutral.

Evaluation

The main comparison uses the same 512 held-out examples for every target. It contains 256 InferBR examples and 256 InferES examples. Sampling uses seed 42. The sample-ID digest is da18f9ec812db66ceb731299b2986d629121334b0565d0d1ea721b25e396d25e. Local results and latency come from four-model-nli-r2. TypeSafe results come from typesafe-comparison-r1 on the same sample IDs.

Model	PT macro F1	ES macro F1	Overall macro F1	Balanced accuracy	Brier	ECE	p50	p95	Peak VRAM
Laya PT/ES NLI	0.9295	0.8028	0.8670	0.8674	0.2185	0.0611	31.7 ms	34.0 ms	0.66 GiB
Laya Multilingual base	0.6054	0.5150	0.5622	0.5583	0.5926	0.1191	31.2 ms	35.0 ms	0.66 GiB
OpenJEV v2	0.9248	0.7943	0.8601	0.8591	0.2057	0.0515	120.4 ms	143.5 ms	9.21 GiB
TypeSafe JEV API	0.9532	0.8366	0.8950	0.8944	0.1518	0.0188	585.6 ms	668.5 ms	not available

ECE uses ten equal-width bins and top-label confidence. Brier is the sum across the three classes. Laya and TypeSafe use localized typed-choice prompts. OpenJEV uses its native premise and hypothesis template.

The complete sealed test contains 3,315 examples. This checkpoint has 0.8657 macro F1 overall, 0.9324 in Portuguese, and 0.7914 in Spanish. Its option-order agreement is 0.9741. The fitted choice temperature is 6.737636.

Latency and memory

The local benchmark ran on a Modal A100-40GB worker. The image used Python 3.12, PyTorch 2.13.0, and Transformers 5.14.1. Each latency value measures one warm batch-size-one call. It excludes model load time.

Model	p50	p95	Peak VRAM	Load time
Laya PT/ES NLI	31.7 ms	34.0 ms	0.66 GiB	54.8 s
Laya Multilingual base	31.2 ms	35.0 ms	0.66 GiB	41.5 s
OpenJEV v2	120.4 ms	143.5 ms	9.21 GiB	23.7 s
TypeSafe JEV API	585.6 ms	668.5 ms	not available	not applicable

TypeSafe latency includes the remote network. It is not an apples-to-apples local-model latency result. Local measurements use 64 examples after two warm-up calls.

Training

Training updated the complete encoder and decision head. It did not use LoRA or an adapter. The objective was RLCD plus cross-entropy. RLCD used four perturbations, log plus spherical reward, and a linear noise schedule from 0.4 to 0.1.

The run trained independent 4-, 8-, and 16-epoch candidates from the same pinned base revision. Validation selected the 8-epoch checkpoint. The selection key was mean PT/ES macro F1, then weakest-language macro F1, then negative NLL.

The selected run used these values:

• batch size: 16
• gradient accumulation: 4
• encoder learning rate: 2.5e-5
• head learning rate: 1e-4
• AdamW weight decay: 0.01
• cosine learning-rate schedule, minimum 1e-6
• gradient clip: 1.0
• BF16 autocast and saved BF16 weights
• gradient checkpointing with non-reentrant execution
• seed: 42
• train, validation, and sealed test sizes: 13,971, 1,275, and 3,315

Portuguese and Spanish had equal sampling weight. Each epoch balanced all six language-label buckets. The fitted validation temperature was saved in rl_agent_config.json.

A Spanish-focused continuation tested 1, 2, and 4 additional epochs with a 2:1 Spanish-to-Portuguese sampling weight. No candidate passed the fixed promotion gate. This release therefore uses the original 8-epoch checkpoint.

Data and license

The Portuguese source is InferBR at revision b6c5e0b27e52477bbc5433d130b0a0e825cf74dc, licensed under MIT. The Spanish source is InferES at revision c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2, licensed under CC BY 4.0.

InferES provides train and test files. The run removed train/test duplicate pairs. It then made a deterministic 10% per-class validation split from the remaining train data with seed 42. InferBR rows marked as low quality were excluded. All splits were deduplicated and checked for pair overlap.

The base checkpoint declares Apache 2.0. This derivative is released under Apache 2.0. InferES attribution remains required. See ATTRIBUTIONS.md and provenance.json. This repository does not redistribute any training dataset or audio.

Limitations

• Spanish is materially weaker than Portuguese on both validation and test data.
• Calibration is domain-specific. Refit temperature on a held-out target-domain set before using probabilities as risk scores.
• The 512-example comparison has no confidence intervals and does not establish superiority over OpenJEV or TypeSafe.
• The checkpoint was trained only for three-way NLI. Other Laya question types were not retrained or validated.
• The local latency results apply to the stated software and GPU. CPU and other GPU results can differ.
• Dataset artifacts can contain annotation errors and social biases from their source corpora.

Files

• model.safetensors: full fine-tuned model weights.
• encoder/ and tokenizer/: pinned model configuration and tokenizer.
• rl_agent_config.json: Laya runtime configuration and calibrated temperature.
• training.json: training configuration, validation metrics, history, and source manifest.
• evaluation_results.json: compact sealed-test and comparison results.
• provenance.json: base revision, dataset revisions, runtime, and compatibility gates.
• SHA256SUMS: release integrity manifest.