shishishi3/laya-deberta-imdb-ensemble

Laya–DeBERTa IMDB sentiment ensemble

A reproducible research and coursework release for English movie-review sentiment classification. It contains two fine-tuned Laya checkpoints, one fine-tuned DeBERTa-v3-large checkpoint, frozen calibration parameters, and the CPU FP32 inference entry point. Predictions are positive or negative, with class probabilities.

• Source and experimental evidence: lliooly/laya-imdb, runtime source snapshot f39da7f9714a76885ad0dbff1133752549e543ed.
• Publisher: shishishi3.
• Weights: three .safetensors files; approximately 3.44 GB including accompanying assets.
• Execution: the supplied Python entry point. Automatic transformers.pipeline() loading and hosted inference are not implemented for the full ensemble.

Quick start

Use Python 3.12 and uv. Download the whole repository so that the relative model and tokenizer paths stay intact:

hf download shishishi3/laya-deberta-imdb-ensemble --local-dir imdb-ensemble
uv run --python 3.12 --with-requirements imdb-ensemble/requirements.txt python imdb-ensemble/predict_final.py --repo imdb-ensemble/runtime --text "The movie was beautifully acted and thoroughly enjoyable."

The first dependency installation requires internet access. The prediction entry point then loads local files and disables model-network access. It performs CPU FP32 inference; no GPU, training or calibration fitting is needed. The full ensemble can take time to load and requires sufficient host memory for the checkpoints; this release has not established a minimum RAM requirement.

To verify the bound runtime source, frozen calibration and model assets without loading the models:

python imdb-ensemble/predict_final.py --repo imdb-ensemble/runtime --verify-only

The output contains label, probabilities, branch probabilities in components, and input truncation information in truncation. Positive probability at least 0.5 yields positive. The entry point accepts one English review through --text.

Method

The Laya branch uses baseline-s43 and combined-s42. Each checkpoint uses frozen temperature scaling and the project's text/question encoding. Prefix and head/tail views are combined for long reviews; short inputs use their native view. The two Laya checkpoints are combined with the persisted coefficient 0.7020326697613102, using the original implementation's ordering.

DeBERTa-v3-large uses the saved head/tail input policy and temperature 1.1374971945707235. The final positive probability is the average of the Laya-branch probability and DeBERTa probability (50:50). No label or newly fitted parameter is used during prediction. fusion.json and frozen-fit.json bind the exact policy and asset hashes.

Data and evaluation

The original official training split was divided into 22,100 gradient-training, 400 calibration, and 2,500 development records. The original fixed system was evaluated on 25,000 official test records. The published checkpoints are that fixed system; they were not selected as the best seed from the later supplementary runs.

Original fixed system	Official test Accuracy
Two-Laya branch	96.344%
DeBERTa-v3-large	96.932%
Fixed 50:50 ensemble	97.028% (24,257 / 25,000)

The ensemble's official-test Macro-F1 is 0.970280, NLL 0.092618, and 15-bin top-label ECE 0.650%. It gains 24 correct predictions over DeBERTa (+0.096 percentage points); the paired exact McNemar test has p=0.1568, so this comparison does not establish a reliable classification improvement.

The recorded test result combines Laya CUDA FP32 cached probabilities with DeBERTa CPU FP32. This release's arbitrary-text demonstration runs all components on CPU FP32. Four frozen development examples (positive/negative and short/long) and one synthetic review were checked locally; cross-backend numerical differences remain possible. This release does not claim an independently rerun 97.028% result for the all-CPU entry point.

Nine later training runs form three complete systems, all evaluated on the same 2,500-record development split. Equal fusion improves Accuracy in two systems by +0.08 percentage points each and decreases it in the third by −0.16; mean Accuracy is 97.08% for both fusion and DeBERTa. Mean ECE decreases from 1.328% to 0.741%. These supplementary checkpoints are not included here, and the official test was not accessed during that supplementary work. Post-hoc weight candidates are exploratory and do not replace this release's frozen 50:50 rule.

Evidence: fixed test report, supplementary runs.

Intended use and limitations

Use for English movie-review demonstrations, coursework, and reproducible comparisons. Other languages, other domains and high-stakes decisions have not been validated.

• Development, calibration and test partitions have historical use; the fixed test evaluation is not the first project-wide blind evaluation.
• The original training/test partitions contain 123 unique cleaned-text overlaps, including 113 overlaps involving actual gradient-training records; the test split also contains 201 extra duplicate records.
• Repeated systems share one development split. They do not constitute 7,500 independent evaluation records.
• Accuracy gains vary with checkpoint combinations and weights. The evidence supports small gains in some configurations and observed probability-quality improvements, with limited generalization claims.
• Long reviews are truncated/viewed according to the saved policies; sarcasm, mixed sentiment and domain shift may cause errors. Output probabilities are model estimates.
• End-to-end performance and dependency installation have not been validated on every operating system. This is a research release with a custom inference entry point.

Files and provenance

models/ contains the three weights, model configs and tokenizers. runtime/scripts/ contains the 20 SHA-256-bound local runtime modules from GitHub. predict_final.py, requirements.txt, fusion.json and frozen-fit.json provide the runnable policy. FILE_MANIFEST.json records payload sizes and SHA-256 values. The manifest excludes its own hash to avoid circularity.

Base revisions and component licenses are documented in LICENSES.md. The release includes no raw IMDB review texts, optimizer states, training-resume packages, official-test prediction arrays, or student ID/class information.

中文说明

这是面向英文电影评论的情感分类研究模型，由两个 Laya 模型和 DeBERTa-v3-large 组成，输出正负类别与概率。请使用随附脚本，完整保留分词器、模型配置和冻结参数。原固定评估的测试准确率为 97.028%；该成绩的后端口径、重复实验结果和数据限制均在上文说明。源码与论文工程见 GitHub 项目。