johanneli/fiorillo-v0.1

Fiorillo v0.1

Fiorillo v0.1 is a medical typed-decision model. I fine-tuned Laya (convaiinnovations/laya), a 421-million-parameter decision model built on ModernBERT-large, to answer two questions with calibrated probabilities in one forward pass: the answer to a biomedical research question from its abstract (PubMedQA: yes, no or maybe), and how soon a patient with diabetes is readmitted after a hospital discharge (UCI Diabetes 130-US hospitals, 1999 to 2008: none, after 30 days, or within 30 days). It is for research. It is not a medical device and must not be used to make clinical decisions.

Other medical models in the Laya family (Hugging Face check, 27 September 2026): Duan (wipen/Duan), licensed CC BY-NC 4.0 for non-commercial use, and impacte/laya-medical-escalation, an emergency severity index (ESI) triage model licensed Apache 2.0.

Correction: the version of this card at revision 0a2e55e (DOI 10.57967/hf/10639) said this was the first openly licensed medical typed-decision model built on Laya that permits commercial use. That sentence rested on a check at 21:23 UTC on 27 September 2026. impacte/laya-medical-escalation was published at 22:15 UTC, before this model was made public at 23:53 UTC, so I removed the sentence.

Named after my violin, which carries the name of Federigo Fiorillo, whose 36 Études, or studies, are where violinists learn to play in tune.

Abbreviations: ECE is expected calibration error (top answer, 15 equal-width bins); AUROC is the area under the receiver operating characteristic curve; UCI is the University of California, Irvine, Machine Learning Repository; MIMIC is the Medical Information Mart for Intensive Care; NRD is the Nationwide Readmissions Database; NIS is the National Inpatient Sample; NHAMCS is the National Hospital Ambulatory Medical Care Survey.

The two questions

The model was trained and evaluated on these definitions, word for word. Other questions are outside what was tested.

Question id	Type	Instructions	Options
pubmedqa_answer	choice	Based only on this abstract, what is the answer to the research question?	yes: the reported results support yes; no: the reported results support no; maybe: the reported results are inconclusive
readmit_window	score	How soon after this hospital discharge is the patient readmitted?	0: no readmission after this discharge; 1: readmitted more than 30 days after this discharge; 2: readmitted within 30 days of this discharge

How to use

Install laya 0.3.20 (pip install laya==0.3.20). The weights ship with every temperature set to 1, so laya returns uncalibrated probabilities until the temperatures in calibration.json are applied. The code below applies them. In laya 0.3.20 the temperature is looked up by question type and number of options, so these two settings calibrate any choice question with 3 to 5 options and any score question with 3 to 5 levels; only the two questions above were calibrated and tested.

import json
from huggingface_hub import hf_hub_download
from laya import Agent

repo = "johanneli/fiorillo-v0.1"
agent = Agent(repo)
temperature = json.load(open(hf_hub_download(repo, "calibration.json")))["temperature"]
agent.temperature_by_options = {"choice:3-5": temperature["pubmedqa_answer"],
                                "score:3-5": temperature["readmit_window"]}

questions = {
    "pubmedqa_answer": {
        "type": "choice",
        "instructions": "Based only on this abstract, what is the answer to the research question?",
        "criteria": {"yes": "the reported results support yes",
                     "no": "the reported results support no",
                     "maybe": "the reported results are inconclusive"}},
    "readmit_window": {
        "type": "score",
        "instructions": "How soon after this hospital discharge is the patient readmitted?",
        "criteria": ["no readmission after this discharge",
                     "readmitted more than 30 days after this discharge",
                     "readmitted within 30 days of this discharge"]},
}

# A PubMedQA-style state: the research question and the abstract's sections, without its conclusion.
abstract = {"question": "Does early mobilization shorten hospital stay after hip fracture surgery?",
            "contexts": [{"section": "BACKGROUND", "text": "..."},
                         {"section": "METHODS", "text": "..."},
                         {"section": "RESULTS", "text": "..."}]}
print(agent.system_one(abstract, {"pubmedqa_answer": questions["pubmedqa_answer"]})["answers"])

# A UCI-style encounter (an invented patient), in the field names and values the model was trained on.
encounter = {"age band": "[70-80)", "sex": "Female", "admission type": "Emergency",
             "admission source": "Emergency Room", "discharge disposition": "Discharged to home",
             "time_in_hospital": 4, "num_lab_procedures": 45, "num_procedures": 1, "num_medications": 18,
             "number_outpatient": 0, "number_emergency": 1, "number_inpatient": 2, "number_diagnoses": 9,
             "diag_1": "428 (Diseases of the circulatory system)",
             "diag_2": "250 (Endocrine, nutritional and metabolic diseases, and immunity disorders)",
             "diag_3": "401 (Diseases of the circulatory system)",
             "max_glu_serum": "None", "A1Cresult": ">8", "medications": {"insulin": "Up"},
             "change": "Ch", "diabetesMed": "Yes"}
print(agent.system_one(encounter, {"readmit_window": questions["readmit_window"]})["answers"])

Each answer carries the probability of every option. Run through this code on a central processing unit (CPU), the released model reproduced its evaluation probabilities to within 0.001 on the six test items I checked.

Training data

Data	License	Used	Splits
PubMedQA (qiaojin/PubMedQA, revision 9001f2853fb87cab8d220904e0de81ac6973b318; official test list at commit 3e7616d288b52edaf38f5fd5c59821c7806bde86)	MIT	The question and the abstract's sections in source order. Left out: the conclusion (long_answer, which states the answer), the label, the original baselines' predictions and MeSH terms. Labels: the expert labels (yes, no, maybe) and, for the automatically labeled items, yes or no	Train 21,050 records: 350 expert-labeled items written 3 times and 20,000 automatically labeled items (yes and no in the expert items' ratio, 0.620 yes). Validation: 150 expert-labeled items (checkpoint choice and temperature) and 2,000 automatically labeled items (monitoring only). Test: the 500 official test items. No PubMed ID appears in two splits
Diabetes 130-US Hospitals for Years 1999 to 2008 (UCI dataset 296)	CC BY 4.0	The first encounter of each patient; encounters ending in death or hospice dropped. Fields: age band, sex, admission type and source, discharge disposition, length of stay, counts of laboratory tests, procedures, medications, prior visits and diagnoses, three diagnoses coded in the International Classification of Diseases, Ninth Revision (ICD-9) with their chapter names, serum glucose and hemoglobin A1c results, diabetes medications and changes to them. Left out: identifiers, race (kept only to report calibration by group), payer code and weight	Split by patient, 80, 10 and 10 percent: 55,978 train, 6,997 validation, 6,998 test

Citations. PubMedQA: Jin Q, Dhingra B, Liu Z, Cohen W, Lu X. PubMedQA: A Dataset for Biomedical Research Question Answering. EMNLP-IJCNLP 2019: 2567 to 2577. UCI data: Clore J, Cios K, DeShazo J, Strack B. Diabetes 130-US Hospitals for Years 1999-2008 [Dataset]. UCI Machine Learning Repository, 2014. https://doi.org/10.24432/C5230J. Strack B, DeShazo JP, Gennings C, Olmo JL, Ventura S, Cios KJ, Clore JN. Impact of HbA1c Measurement on Hospital Readmission Rates: Analysis of 70,000 Clinical Database Patient Records. BioMed Research International 2014: 781670.

No MIMIC, NRD, NIS or NHAMCS data were used to train this model. Laya's model card does not list PubMedQA or the UCI data among its training data (read 27 September 2026).

Training

I started from Laya (convaiinnovations/laya, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851), its encoder and its decision head, and trained both on the two questions together.

• Loss: log loss over the options. AdamW (encoder learning rate 2.5e-5, head 1.0e-4, weight decay 0.01), cosine schedule, gradient clipping at 1.0, float16 mixed precision, gradient checkpointing.
• One epoch, 2,408 optimizer steps, 32 sequences a step (8 per micro-batch, 4 accumulated), each micro-batch from one question; states up to 768 tokens, decision head 192.
• Checkpoint: validation every 250 steps; I kept the step with the lowest mean of the two questions' validation log losses (step 1,500).
• Calibration: one temperature per question, fitted on the validation items and shipped in calibration.json: PubMedQA 1.726, readmission 1.1111.
• Hardware: one NVIDIA H100 PCIe; 2,329 seconds of training.

Development history. This is the third training run, all on 27 September 2026. Run 1 wrote each expert-labeled PubMedQA training item 10 times and checked validation every 1,000 steps; the model memorized those items. Run 2 wrote them 3 times and checked every 250 steps, but its PubMedQA checkpoint choice and temperature came from 50 validation items and were poorly determined (the temperature's bootstrap 95% interval ran from 1.43 to 5.64). For run 3 I moved 100 expert-labeled items from training to validation (150 in validation, 350 in training) and changed nothing else. Each change to this model's training was written down, with reasons drawn from training logs and validation data, before the run it applied to. The test split was first scored in run 1, so these results are not from a test split used once; every run's test results are kept, and run 2's appear below as "Earlier run of this model".

Evaluation

All results are on the test splits: 500 PubMedQA items (445 of them yes or no) and 6,998 readmission encounters. Intervals are 95% paired percentile bootstrap intervals (1,000 resamples, seed 20260927). Each neural model has one temperature per question fitted on the validation split; the prior and LightGBM are used as they are. In the comparisons with Duan, Duan is scored with and without its temperature and the better of the two is used for each measure, a rule set in the evaluation code before Duan was run. Systems:

• This model, and the earlier run of it (run 2).
• Two research arms from run 2, to compare readers: each has a new, randomly initialized decision head, trained alone for the first 250 steps and then with the reader, on run 2's data (50 PubMedQA validation items, 450 expert-labeled training items). One uses Laya's own reader; the other uses the reader of BioClinical ModernBERT-large (thomas-sounack/BioClinical-ModernBERT-large, whose pretraining included MIMIC clinical notes). Neither is released.
• Laya as published, not fine-tuned.
• LightGBM (gradient-boosted trees) on the same UCI fields, readmission only.
• Duan (wipen/Duan, revision 50e0615b576e4829cbad5c193d42edd0cadf2784), a Laya-family medical model licensed for non-commercial use, evaluated only: in its own question wording and input format, and in this model's wording. Duan's PubMedQA question is yes or no, so it is scored on the 445 yes or no items.
• The prior: every item gets the training split's label frequencies.

PubMedQA, three answers (500 items)

System	n	Log loss	ECE	Macro-F1	Accuracy
Fiorillo v0.1 (this model)	500	0.7428 (0.6698 to 0.8164)	0.0276 (0.0301 to 0.0818)	0.5005 (0.4744 to 0.5258)	0.7140 (0.6720 to 0.7560)
Earlier run of this model (run 2)	500	0.8091 (0.7618 to 0.8557)	0.1310 (0.0982 to 0.1703)	0.4859 (0.4597 to 0.5113)	0.6980 (0.6580 to 0.7380)
Laya reader, new decision head	500	0.7943 (0.7351 to 0.8533)	0.0600 (0.0443 to 0.1063)	0.4786 (0.4471 to 0.5132)	0.6720 (0.6320 to 0.7120)
BioClinical ModernBERT reader, new decision head	500	0.7051 (0.6369 to 0.7745)	0.0489 (0.0366 to 0.0940)	0.5140 (0.4877 to 0.5389)	0.7360 (0.6920 to 0.7760)
Laya, not fine-tuned	500	1.0431 (1.0207 to 1.0674)	0.0797 (0.0440 to 0.1208)	0.2889 (0.2518 to 0.3267)	0.5140 (0.4700 to 0.5520)
Duan, this model's wording	500	1.1398 (1.0432 to 1.2413)	0.2321 (0.1981 to 0.2786)	0.4205 (0.3738 to 0.4682)	0.5680 (0.5240 to 0.6080)
Prior (training label frequencies)	500	1.1593 (1.0401 to 1.2920)	0.0646 (0.0226 to 0.1086)	0.2371 (0.2246 to 0.2484)	0.5520 (0.5080 to 0.5940)

Resampling inflates ECE, so some ECE intervals lie above their own estimate (see Calibration).

This model never gave maybe as its top answer (0 of 500 test items; 55 items are maybe), which holds its three-answer macro-F1 down. No system named more than 6 of the 55 maybe items correctly.

PubMedQA, yes or no items (445 items; maybe items removed and the other two probabilities rescaled to sum to 1)

System	n	Log loss	ECE	Macro-F1	Accuracy
Fiorillo v0.1 (this model)	445	0.4412 (0.3966 to 0.4839)	0.0439 (0.0329 to 0.0850)	0.7977 (0.7597 to 0.8329)	0.8022 (0.7640 to 0.8382)
Earlier run of this model (run 2)	445	0.5211 (0.4992 to 0.5457)	0.1313 (0.1029 to 0.1653)	0.7735 (0.7331 to 0.8100)	0.7843 (0.7461 to 0.8202)
Laya reader, new decision head	445	0.4983 (0.4617 to 0.5380)	0.0570 (0.0434 to 0.1024)	0.7441 (0.7044 to 0.7840)	0.7551 (0.7190 to 0.7933)
BioClinical ModernBERT reader, new decision head	445	0.4046 (0.3620 to 0.4528)	0.0397 (0.0293 to 0.0787)	0.8192 (0.7812 to 0.8524)	0.8270 (0.7910 to 0.8584)
Laya, not fine-tuned	445	0.6467 (0.6220 to 0.6734)	0.0391 (0.0206 to 0.0861)	0.4550 (0.4145 to 0.4958)	0.6382 (0.5933 to 0.6831)
Duan, its own wording and input format	445	1.5034 (1.3245 to 1.6887)	0.3127 (0.2734 to 0.3591)	0.6089 (0.5625 to 0.6549)	0.6337 (0.5910 to 0.6787)
Duan, this model's wording	445	0.8393 (0.7404 to 0.9318)	0.2088 (0.1734 to 0.2544)	0.6115 (0.5668 to 0.6554)	0.6494 (0.6067 to 0.6944)
Prior (training label frequencies)	445	0.6640 (0.6419 to 0.6860)	0.0001 (0.0001 to 0.0516)	0.3828 (0.3652 to 0.3995)	0.6202 (0.5753 to 0.6652)

Readmission window (6,998 encounters)

Macro-F1 with decision weights: before scoring, each system's decision is tuned on the validation split by weighting the probability of each level (level 0 fixed at 1; levels 1 and 2 on a grid from 1/4 to 16), because readmission within 30 days is only 8.7% of test encounters and a calibrated model seldom makes it the top answer. AUROC is for readmission within 30 days against the rest.

System	n	Log loss	ECE	Accuracy	Macro-F1	Macro-F1, decision weights	AUROC, within 30 days
Fiorillo v0.1 (this model)	6,998	0.8524 (0.8384 to 0.8662)	0.0169 (0.0101 to 0.0295)	0.6045 (0.5935 to 0.6160)	0.2934 (0.2863 to 0.3002)	0.4261 (0.4125 to 0.4409)	0.6423 (0.6185 to 0.6661)
Earlier run of this model (run 2)	6,998	0.8700 (0.8571 to 0.8828)	0.0283 (0.0198 to 0.0400)	0.5942 (0.5829 to 0.6047)	0.2485 (0.2455 to 0.2512)	0.4090 (0.3971 to 0.4208)	0.6169 (0.5912 to 0.6426)
Laya reader, new decision head	6,998	0.8829 (0.8705 to 0.8951)	0.0288 (0.0181 to 0.0403)	0.5942 (0.5829 to 0.6047)	0.2485 (0.2455 to 0.2512)	0.3656 (0.3534 to 0.3788)	0.5855 (0.5607 to 0.6110)
BioClinical ModernBERT reader, new decision head	6,998	0.8523 (0.8386 to 0.8657)	0.0187 (0.0131 to 0.0322)	0.6059 (0.5940 to 0.6173)	0.3372 (0.3285 to 0.3459)	0.4168 (0.4038 to 0.4297)	0.6343 (0.6128 to 0.6583)
Laya, not fine-tuned	6,998	1.1289 (1.1277 to 1.1300)	0.2207 (0.2125 to 0.2285)	0.1296 (0.1219 to 0.1379)	0.1312 (0.1228 to 0.1395)	0.3138 (0.3054 to 0.3230)	0.5099 (0.4868 to 0.5330)
LightGBM on the same fields	6,998	0.8400 (0.8254 to 0.8538)	0.0066 (0.0071 to 0.0214)	0.6096 (0.5975 to 0.6216)	0.3494 (0.3396 to 0.3593)	0.4417 (0.4285 to 0.4564)	0.6469 (0.6226 to 0.6715)
Duan, its own wording and input format	6,998	0.8812 (0.8666 to 0.8954)	0.0348 (0.0261 to 0.0453)	0.6042 (0.5930 to 0.6149)	0.3525 (0.3423 to 0.3618)	0.4184 (0.4070 to 0.4311)	0.6171 (0.5930 to 0.6400)
Duan, this model's wording	6,998	0.9507 (0.9374 to 0.9642)	0.0905 (0.0794 to 0.1022)	0.5976 (0.5862 to 0.6090)	0.3467 (0.3378 to 0.3560)	0.3680 (0.3570 to 0.3787)	0.5558 (0.5299 to 0.5775)
Prior (training label frequencies)	6,998	0.8861 (0.8734 to 0.8984)	0.0017 (0.0002 to 0.0127)	0.5942 (0.5829 to 0.6047)	0.2485 (0.2455 to 0.2512)	0.2485 (0.2455 to 0.2512)	0.5000 (0.5000 to 0.5000)

System	n	Calibration intercept, within 30 days	Calibration slope, within 30 days	Quadratic weighted kappa
Fiorillo v0.1 (this model)	6,998	-0.0957 (-0.1781 to -0.0103)	1.1883 (1.0064 to 1.3756)	0.0537 (0.0422 to 0.0657)
Earlier run of this model (run 2)	6,998	-0.1801 (-0.2632 to -0.0976)	1.3139 (1.0701 to 1.5511)	0.0000 (0.0000 to 0.0000)
Laya reader, new decision head	6,998	-0.1137 (-0.1956 to -0.0300)	5.7489 (4.1085 to 7.5109)	0.0000 (0.0000 to 0.0000)
BioClinical ModernBERT reader, new decision head	6,998	-0.0308 (-0.1136 to 0.0546)	0.9412 (0.7822 to 1.1087)	0.0998 (0.0823 to 0.1164)
Laya, not fine-tuned	6,998	-1.7309 (-1.8136 to -1.6468)	1.0256 (-1.6275 to 3.6682)	-0.0033 (-0.0086 to 0.0018)
LightGBM on the same fields	6,998	-0.0306 (-0.1171 to 0.0549)	0.9781 (0.8240 to 1.1292)	0.1081 (0.0912 to 0.1262)
Duan, its own wording and input format	6,998	-0.5513 (-0.6350 to -0.4656)	0.7522 (0.5280 to 0.9873)	0.1039 (0.0868 to 0.1213)
Duan, this model's wording	6,998	-0.8994 (-0.9828 to -0.8150)	0.1427 (-0.0392 to 0.3215)	0.0990 (0.0809 to 0.1181)
Prior (training label frequencies)	6,998	-0.0299 (-0.1120 to 0.0537)	not defined	0.0000 (0.0000 to 0.0000)

Paired comparisons with this model

Differences are this model minus the reference; a negative difference in log loss or ECE favors this model.

Comparison	Question	Measure	This model	Reference	Difference (95% interval)	Verdict
Prior	PubMedQA, three answers	Log loss	0.7428	1.1593	-0.4165 (-0.4939 to -0.3395)	Significant, better
Prior	Readmission	Log loss	0.8524	0.8861	-0.0337 (-0.0390 to -0.0284)	Significant, better
Duan, its own format	PubMedQA, yes or no items	Macro-F1	0.7977	0.6089	0.1887 (0.1334 to 0.2454)	Significant, better
Duan, its own format	PubMedQA, yes or no items	Log loss	0.4412	1.5034	-1.0622 (-1.2466 to -0.8818)	Significant, better
Duan, its own format	PubMedQA, yes or no items	ECE	0.0439	0.3127	-0.2688 (-0.3096 to -0.2100)	Significant, better
Duan, its own format	Readmission	Macro-F1, decision weights	0.4261	0.4184	0.0076 (-0.0072 to 0.0210)	No difference detected
Duan, its own format	Readmission	Macro-F1	0.2934	0.3525	-0.0591 (-0.0684 to -0.0501)	Significant, worse
Duan, its own format	Readmission	Quadratic weighted kappa	0.0537	0.1039	-0.0503 (-0.0659 to -0.0339)	Significant, worse
Duan, its own format	Readmission	Log loss	0.8524	0.8812	-0.0288 (-0.0382 to -0.0199)	Significant, better
Duan, its own format	Readmission	ECE	0.0169	0.0348	-0.0179 (-0.0312 to 0.0003)	No difference detected
Duan, its own format	Readmission	AUROC, within 30 days	0.6423	0.6171	0.0253 (0.0039 to 0.0464)	Significant, better
LightGBM	Readmission	AUROC, within 30 days	0.6423	0.6469	-0.0045 (-0.0253 to 0.0158)	No difference detected
LightGBM	Readmission	Log loss	0.8524	0.8400	0.0124 (0.0073 to 0.0175)	Significant, worse
LightGBM	Readmission	ECE	0.0169	0.0066	0.0103 (-0.0074 to 0.0188)	No difference detected

Duan in this model's wording (sensitivity analysis):

Question	Measure	This model	Duan, this model's wording	Difference (95% interval)	Verdict
PubMedQA, yes or no items	Macro-F1	0.7977	0.6115	0.1861 (0.1291 to 0.2382)	Significant, better
PubMedQA, yes or no items	Log loss	0.4412	0.8393	-0.3981 (-0.4938 to -0.2949)	Significant, better
PubMedQA, yes or no items	ECE	0.0439	0.2088	-0.1649 (-0.2065 to -0.1061)	Significant, better
Readmission	Macro-F1	0.2934	0.3467	-0.0533 (-0.0621 to -0.0450)	Significant, worse
Readmission	Macro-F1, decision weights	0.4261	0.3705	0.0555 (0.0408 to 0.0707)	Significant, better
Readmission	Quadratic weighted kappa	0.0537	0.0990	-0.0453 (-0.0630 to -0.0299)	Significant, worse
Readmission	Log loss	0.8524	0.9507	-0.0983 (-0.1085 to -0.0877)	Significant, better
Readmission	ECE	0.0169	0.0905	-0.0736 (-0.0880 to -0.0532)	Significant, better
Readmission	AUROC, within 30 days	0.6423	0.5558	0.0866 (0.0594 to 0.1153)	Significant, better

What the reader and the decision head contribute (research arms, not released):

Comparison	Question	Measure	First	Second	Difference (95% interval)	Verdict
BioClinical reader minus Laya reader, both with a new head	PubMedQA, three answers	Accuracy	0.7360	0.6720	0.0640 (0.0300 to 0.1020)	Significant, better
BioClinical reader minus Laya reader, both with a new head	PubMedQA, three answers	Log loss	0.7051	0.7943	-0.0892 (-0.1350 to -0.0429)	Significant, better
BioClinical reader minus Laya reader, both with a new head	PubMedQA, three answers	ECE	0.0489	0.0600	-0.0111 (-0.0507 to 0.0320)	No difference detected
BioClinical reader minus Laya reader, both with a new head	Readmission	AUROC, within 30 days	0.6343	0.5855	0.0489 (0.0289 to 0.0682)	Significant, better
BioClinical reader minus Laya reader, both with a new head	Readmission	Log loss	0.8523	0.8829	-0.0306 (-0.0364 to -0.0251)	Significant, better
BioClinical reader minus Laya reader, both with a new head	Readmission	ECE	0.0187	0.0288	-0.0102 (-0.0218 to 0.0065)	No difference detected
Laya's own head minus a new head, both on Laya's reader (run 2)	PubMedQA, three answers	Accuracy	0.6980	0.6720	0.0260 (-0.0080 to 0.0600)	No difference detected
Laya's own head minus a new head, both on Laya's reader (run 2)	PubMedQA, three answers	Log loss	0.8091	0.7943	0.0148 (-0.0148 to 0.0448)	No difference detected
Laya's own head minus a new head, both on Laya's reader (run 2)	PubMedQA, three answers	ECE	0.1310	0.0600	0.0710 (0.0186 to 0.1019)	Significant, worse
Laya's own head minus a new head, both on Laya's reader (run 2)	Readmission	AUROC, within 30 days	0.6169	0.5855	0.0315 (0.0140 to 0.0485)	Significant, better
Laya's own head minus a new head, both on Laya's reader (run 2)	Readmission	Log loss	0.8700	0.8829	-0.0129 (-0.0160 to -0.0096)	Significant, better
Laya's own head minus a new head, both on Laya's reader (run 2)	Readmission	ECE	0.0283	0.0288	-0.0005 (-0.0091 to 0.0101)	No difference detected

Release criteria

These three criteria were fixed before training. The model is released because all three passed on the test split.

Criterion (fixed before training)	Result	Values
1. Expected calibration error at most 0.05 on both questions	PASS	PubMedQA 0.0276; readmission 0.0169
2. Log loss below the prior's on both questions, 95% interval of the difference excluding zero	PASS	PubMedQA 0.7428 against 1.1593, difference -0.4165 (-0.4939 to -0.3395); readmission 0.8524 against 0.8861, difference -0.0337 (-0.0390 to -0.0284)
3. Macro-F1 not below Duan's point estimate (PubMedQA yes or no items, top answer; readmission, decision weights)	PASS	PubMedQA 0.7977 against 0.6089; readmission 0.4261 against 0.4184

The readmission part of criterion 3 passed on the point estimate; the difference from Duan, 0.0076 (-0.0072 to 0.0210), is not significant, and its interval excludes Duan being better by more than 0.0072.

Calibration

The temperature lowered PubMedQA log loss from 0.8749 to 0.7428 and ECE from 0.1371 to 0.0276; for readmission, log loss from 0.8531 to 0.8524 and ECE from 0.0303 to 0.0169. For readmission within 30 days, the calibration intercept is -0.0957 (-0.1781 to -0.0103) and the slope is 1.1883 (1.0064 to 1.3756) (1 is ideal): the model predicts slightly more 30-day readmissions than occur, and its predictions are slightly too close together.

Resampling inflates ECE, so an ECE interval can lie above its own estimate. The ECE on 500 items is also noisy: on validation data, a model calibrated by construction with this model's confidence profile would show an ECE above 0.05 on 500 items about 23% of the time.

Calibration for readmission within 30 days by group, for groups with at least 100 test encounters:

Group	n	Observed within 30 days	Expected within 30 days	Observed / expected	ECE
Sex: Female	3,729	0.0880 (0.0791 to 0.0971)	0.0964 (0.0949 to 0.0979)	0.913 (0.823 to 1.006)	0.0212 (0.0146 to 0.0388)
Sex: Male	3,269	0.0857 (0.0765 to 0.0954)	0.0926 (0.0913 to 0.0940)	0.925 (0.826 to 1.026)	0.0194 (0.0121 to 0.0373)
Age: [20-30)	110	0.0545 (0.0182 to 0.1000)	0.0683 (0.0623 to 0.0754)	0.799 (0.263 to 1.425)	0.0896 (0.0678 to 0.1883)
Age: [30-40)	255	0.0824 (0.0510 to 0.1137)	0.0715 (0.0676 to 0.0758)	1.151 (0.714 to 1.605)	0.0543 (0.0396 to 0.1162)
Age: [40-50)	726	0.0565 (0.0413 to 0.0730)	0.0714 (0.0692 to 0.0737)	0.791 (0.570 to 1.022)	0.0292 (0.0223 to 0.0671)
Age: [50-60)	1,152	0.0677 (0.0538 to 0.0825)	0.0786 (0.0766 to 0.0808)	0.861 (0.680 to 1.036)	0.0234 (0.0182 to 0.0573)
Age: [60-70)	1,605	0.0903 (0.0766 to 0.1047)	0.0917 (0.0900 to 0.0934)	0.985 (0.837 to 1.136)	0.0407 (0.0273 to 0.0674)
Age: [70-80)	1,802	0.0932 (0.0805 to 0.1060)	0.1074 (0.1056 to 0.1091)	0.868 (0.754 to 0.985)	0.0211 (0.0151 to 0.0493)
Age: [80-90)	1,088	0.1204 (0.1020 to 0.1388)	0.1174 (0.1150 to 0.1196)	1.026 (0.873 to 1.185)	0.0300 (0.0229 to 0.0638)
Age: [90-100)	184	0.0924 (0.0543 to 0.1413)	0.1128 (0.1072 to 0.1184)	0.819 (0.477 to 1.235)	0.0651 (0.0396 to 0.1429)
Race: African American	1,214	0.0832 (0.0684 to 0.0997)	0.0890 (0.0870 to 0.0913)	0.935 (0.772 to 1.127)	0.0375 (0.0232 to 0.0645)
Race: Caucasian	5,293	0.0886 (0.0805 to 0.0964)	0.0963 (0.0952 to 0.0973)	0.921 (0.835 to 0.998)	0.0192 (0.0109 to 0.0336)
Race: Hispanic	137	0.0803 (0.0438 to 0.1314)	0.0935 (0.0864 to 0.1009)	0.859 (0.447 to 1.394)	0.1151 (0.0791 to 0.2065)
Race: Other	110	0.0455 (0.0091 to 0.0818)	0.0832 (0.0777 to 0.0898)	0.546 (0.113 to 1.007)	0.0951 (0.0565 to 0.1915)
Race: unknown	194	0.0773 (0.0412 to 0.1186)	0.0928 (0.0873 to 0.0991)	0.834 (0.448 to 1.234)	0.1184 (0.0875 to 0.1852)

Known contamination

PubMedQA contexts are PubMed abstracts that PubMed-trained readers may have seen with their conclusions; Duan trained on 445 expert-labeled PubMedQA items that may overlap the official test list, and on a random 20,000-encounter UCI sample that may overlap the test split used here. Both favor Duan.

Limitations

• Research use only. It is not a medical device, has not been validated prospectively or at any other site, and must not guide care.
• Two questions only. Other questions, wordings or input formats are untested and uncalibrated.
• The readmission data are from 130 US hospitals between 1999 and 2008, first encounters only, and practice has changed since. On these data, LightGBM on the same fields had lower log loss (0.8400 against 0.8524), and no difference was detected in AUROC.
• The model never gave maybe as its top answer on the test split, and cannot yet tell maybe abstracts apart from the others.
• The PubMedQA temperature was fitted on 150 items; its bootstrap 95% interval runs from 1.37 to 2.22.
• The test split was used by all three training runs (see Development history).
• English only.

Lineage

ModernBERT-large (answerdotai/ModernBERT-large, Apache 2.0) was fine-tuned into Laya (convaiinnovations/laya, Apache 2.0), which was fine-tuned into this model.

License and citation

Apache 2.0. The training data keep their own licenses (PubMedQA MIT; UCI data CC BY 4.0); the NOTICE file carries the attributions.

Li JE. Fiorillo v0.1: a calibrated medical typed-decision model built on Laya. Hugging Face, 2026. https://huggingface.co/johanneli/fiorillo-v0.1