femto-jev

7.7M parameters, 30 MB in fp32. A decision model that reads text or speech: one state and a set of typed questions in, a probability for every option out, in one forward pass per question. No text is generated, so there is nothing to parse.

Speech goes through femto-asr (2.39M parameters), so the whole spoken pipeline is 10.1M parameters.

It will not beat a 9B-parameter model at reasoning, and a 4.7B audio model hears far better. It is for the case where the decision has to be made on a small machine.

What it answers

Three question types, the same shape as typed-decisions, Laya and Oscar-1:

type	you give	you get
choice	options, each with an optional description	one probability per option
score	ordered rubric levels	one probability per level
noul	a yes/no statement	P(false), P(true)

The questions are plain text written at call time. Nothing is retrained when you change them.

A recording can be read three ways. All three come from the audio alone:

• transcript + frames (joint, the default): femto-asr's transcript, followed by its hidden frames.
• transcript (hyp): the transcript only, a classic cascade.
• frames (audio): no words at all, only femto-asr's frames.

Results

All numbers are accuracy with the typed-decisions scorer, measured by us with one evaluator for every model in a table.

typed-decisions (text)

Official test split, 400 cases / 2,000 decisions.

model	parameters	accuracy
laya-typed-decisions	421M	0.767
Oscar-1-68M	74.7M	0.742
femto-jev	7.7M	0.732
JEV-9B	9.0B	0.706
Oscar-1-32M	35.6M	0.699
Oscar-1-17M	18.4M	0.682
laya (never trained on it)	421M	0.360

This benchmark is in-distribution for femto-jev, the Oscars and laya-typed-decisions: all trained on its training split. Read it as what each size learned, not as general ability. No test case is in femto-jev's training data (0 of 400 test states).

Speech

Each set pools its native question with a yes/no question. SLURP is real speech and its training split is in the training data. The four TTS sets are Kokoro voices held out of training. MInDS-14 is real 8 kHz phone speech that was never trained on.

test set	true words	transcript	frames	transcript + frames	AudioJev (4.7B, raw audio)
SLURP (real, 12,840 clips)	0.916	0.586	0.692	0.714	0.887
Banking77 (TTS)	0.915	0.872	0.786	0.893	0.878
Emotion (TTS)	0.929	0.898	0.627	0.902	0.691
SST-5 (TTS)	0.587	0.567	0.448	0.570	0.717
AG News (TTS)	0.896	0.855	0.687	0.854	0.845
MInDS-14 en (real, never trained on)	0.781	0.561	0.369	0.558	0.907

• "True words" is the same model reading the correct text: the ceiling a perfect recogniser would give.
• On SLURP the frames add 13 points to the transcript. femto-asr's word error rate is high on SLURP's far-field recordings, and the frames carry what the transcript lost.
• On MInDS-14 they add nothing. The frames only help on speech like the training speech. On new audio conditions, expect the transcript number.
• AudioJev hears far better on real speech at about 460 times the size. SLURP is in its training data too.

The same transcript, read by other models

A cascade is only as good as its reader, so here every model gets the identical femto-asr transcript. The first five sets are in femto-jev's training mix and zero-shot for the others; MInDS-14 is zero-shot for all.

model	parameters	SLURP	Banking77	Emotion	SST-5	AG News	MInDS-14
femto-jev, transcript + frames	7.7M	0.714	0.893	0.902	0.570	0.854	0.558
femto-jev, transcript only	7.7M	0.586	0.872	0.898	0.567	0.855	0.561
Oscar-1-17M	18.4M	0.223	0.458	0.481	0.418	0.393	0.401
Oscar-1-32M	35.6M	0.244	0.432	0.557	0.575	0.516	0.422
Oscar-1-68M	74.7M	0.242	0.586	0.881	0.529	0.819	0.491
laya	421M	0.326	0.752	0.593	0.553	0.847	0.652
laya-typed-decisions	421M	0.334	0.757	0.654	0.664	0.856	0.649
JEV-9B	9.0B	0.451	0.828	0.702	0.666	0.859	0.724

JEV-9B's SLURP number covers two of the three questions (it takes at most 16 options); on those two, femto-jev scores 0.761 with transcript + frames and 0.648 with the transcript. On speech it has not seen (MInDS-14), the larger readers recover more from the same noisy transcript.

Text sets it never trained on

model	parameters	tweet sentiment	tweet offensive	tweet irony	MInDS-14, true words	MInDS-14, femto-asr transcript
femto-jev	7.7M	0.521	0.411	0.589	0.781	0.561
Oscar-1-17M	18.4M	0.483	0.713	0.603	0.602	0.401
Oscar-1-32M	35.6M	0.486	0.713	0.598	0.640	0.422
Oscar-1-68M	74.7M	0.573	0.722	0.586	0.750	0.491
laya	421M	0.578	0.765	0.725	0.901	0.652
laya-typed-decisions	421M	0.619	0.788	0.686	0.894	0.649
JEV-9B	9.0B	0.553	0.773	0.811	0.933	0.724

Tweets are tweet_eval test splits. Tweet-offensive is a real weakness: always answering "not offensive" scores about 0.72, and femto-jev is well below that.

Text sets it trained on

From the true words, with our question wording (zero-shot for the other models, and the small Oscars do not follow this wording well):

model	SLURP	Banking77	Emotion	SST-5	AG News
femto-jev	0.916	0.915	0.929	0.587	0.896
Oscar-1-17M	0.307	0.498	0.496	0.464	0.437
Oscar-1-32M	0.328	0.462	0.577	0.614	0.703
Oscar-1-68M	0.432	0.666	0.924	0.548	0.909
laya	0.699	0.813	0.611	0.583	0.907
laya-typed-decisions	0.703	0.819	0.672	0.695	0.907
JEV-9B	0.876	0.877	0.714	0.720	0.882

JEV-9B's SLURP number leaves out the 18-option scenario question (it takes at most 16 options). Other sets in the training mix: DBpedia 0.977, CLINC150 0.950, SNLI 0.771, GoEmotions 0.745, PAWS 0.715, VitaminC 0.658, BoolQ 0.635, WANLI 0.625, STS-B 0.415. Paraphrase, reading comprehension and inference stay close to chance at this size.

Examples

A text

questions = {
    "intent": {"type": "choice", "instructions": "What does the customer want?",
               "criteria": {"refund": "money back for a charge",
                            "cancel": "cancel a card, order or subscription",
                            "balance": "check an account balance", "other": "anything else"}},
    "urgency": {"type": "score", "instructions": "Rate urgency 0 to 3.",
                "criteria": ["routine", "low", "high", "critical"]},
    "angry": {"type": "noul", "instructions": "Is the customer angry?"},
}
jev(questions, text="I was charged twice for the same order, please give me my money back.")

intent   refund 1.00   cancel 0.00   balance 0.00   other 0.00
urgency  routine 0.21  low 0.21  high 0.39  critical 0.20
angry    false 0.77    true 0.23

A spoken command, with your own command list

commands = {"command": {"type": "choice", "instructions": "Which command did the user give?",
    "criteria": {"lights_on": "turn the lights on", "lights_off": "turn the lights off",
                 "timer": "set a timer or an alarm", "music": "play music or a song",
                 "weather": "ask about the weather", "volume": "change the volume",
                 "robot_dance": "make the robot dance", "none": "none of these commands"}}}
r = jev(commands, audio=wav)      # a synthetic voice saying "set a timer for ten minutes"
r["transcript"]                   # 'Set a time er for ten minutes'
r["answers"]["command"]           # timer 1.00

What it did with ten sentences, two synthetic voices each (20 clips, a first look and not a benchmark): 17 right with transcript + frames, 17 with the transcript, 10 with frames alone.

spoken	femto-asr heard	answer
turn on the kitchen lights	Turn on the kitchen lights	lights_on 0.80
switch the lights off please	Witch the lights off pse	lights_off 0.99
play some jazz	Plaplace some jts	music 0.96
will it rain tomorrow	Will it reign to morrow	weather 1.00
wake me up at seven	Wake me up at seven	timer 1.00
it is too dark in here	It is too dark and here	lights_on 0.41
robot, do a little dance	Robt do a little dance	robot_dance 0.60
what is the capital of France	What is the capital of fds	volume 0.79 (wrong: should be none)

It survives a garbled transcript and indirect phrasing. It does not reliably say "none of these", so do not use the confidence to reject out-of-list speech.

Architecture

 text state                          spoken state (16 kHz mono)
     │                                      │
     │                               femto-asr, frozen (2.39M)
     │                                 ├─ greedy transcript ────────┐
     │                                 └─ frames 25 fps × 128       │
     │                                      │ 2 stacked → 12.5/s    │
     │                                      ▼                       │
     │                               Linear 256 → 256               │
     ▼                                      ▼                       ▼
 [CLS] <type> question: … [SEP] [MASK] opt0 [MASK] opt1 … [SEP]  state  [SEP]
                                  │           │                (tokens, frames, or
                                  │           │                 tokens [SEP] frames)
╔═════════════════════════════════╪═══════════╪═══════════════════════════════╗
║  Ettin-17M encoder (ModernBERT, 7 layers, d 256), fully fine-tuned          ║
║  token table shrunk 50,368 → 8,192 rows                                     ║
╚═════════════════════════════════╪═══════════╪═══════════════════════════════╝
                                  ▼           ▼
            2 transformer layers + question-type embedding
                                  ▼           ▼
                       scorer at each option's [MASK]
                                  ▼
                   softmax / temperature  →  P(option)

part	parameters
encoder blocks	3.90M
token embeddings (8,192 rows)	2.10M
decision head	1.65M
speech projector	0.07M
total	7.71M

Three choices carry the model:

• A shrunk vocabulary, not a cut one. Ettin's byte-level BPE is pruned to the 8,192 most useful tokens with every merge parent and byte token kept, so any text still encodes, with no unknown tokens. Training text gets 3% longer and the model goes from 18.5M to 7.7M parameters. It costs about 0.7 points on typed-decisions.
• Speech enters as frames, at the state's positions. The question and options stay text, so spoken input is as open-ended as written input. Every femto-asr frame is kept, two stacked per position.
• Transcript and frames together. One sequence holds both. On real speech this beats either alone (SLURP: 0.586 transcript, 0.692 frames, 0.714 both).

Training

From the released ettin-encoder-17m, with a fresh decision head. No other decision model's weights are used. 1.38M training items, 4 epochs, batch 64, one L40S, 1 h 19 min. The recipe is Laya's RLCD (REINFORCE over noisy logits with a proper scoring rule, plus soft cross-entropy), encoder learning rate 2.5e-5, head 1e-4.

Text. typed-decisions train (1,200 cases); AG News, Emotion, SST-5, Banking77, MNLI; SNLI, WANLI, VitaminC, BoolQ, PAWS, STS-B, CLINC150, GoEmotions, DBpedia. Up to 15,000 rows per source, each with its native question and usually a yes/no variant.

Speech. SLURP train (50.6k real recordings), and Banking77, Emotion, SST-5, AG News, CLINC150, GoEmotions and DBpedia rows spoken by 12 Kokoro-82M voices. Every spoken row trains three ways: frames, transcript, and both.

The one thing that moved typed-decisions. Each typed-decisions training case is shown 16 times per epoch. With each case shown once, the same data gives 0.610; shown 16 times, 0.732; shown 64 times, 0.708. Adding nine more datasets without this changed nothing. Oscar-1's corpus is one third typed-decisions variants, which is the same lever.

Usage

pip install torch transformers safetensors huggingface_hub soundfile

import sys
import soundfile as sf
from huggingface_hub import snapshot_download

REPO = "igorktech/femto-jev"
sys.path.insert(0, snapshot_download(REPO))
from femto_jev import FemtoJev

jev = FemtoJev(REPO)                        # CPU by default; device="cuda" works

r = jev(questions, text="I was charged twice, please return the money.")
print(r["answers"]["intent"])               # {'refund': 0.99…, 'cancel': …}

wav, sr = sf.read("clip.wav", dtype="float32")     # 16 kHz mono
r = jev(questions, audio=wav)               # mode="joint" (default) | "hyp" | "audio"
print(r["transcript"], r["answers"])

From the command line, inside the downloaded folder:

python femto_jev.py --questions q.json --wav clip.wav

A text state can also be a JSON object (an invoice, a ticket), as in typed-decisions. femto-asr is downloaded on the first spoken call and loads with trust_remote_code=True, because it is a custom architecture.

Limitations

• English only.
• Speech is as good as the recogniser. femto-asr is a 2.39M-parameter model trained on audiobooks. On far-field or phone speech its transcripts are poor, and accuracy from audio is about 20 points below accuracy from the true words (1 to 4 on clean synthetic speech).
• Frames alone are weak, and do not transfer. 0.37 on phone speech it never saw. Use the default transcript + frames.
• No reject option you can trust. It rarely answers "none of these", and it can be confidently wrong off its training distribution. Do not gate on confidence without checking on your own data.
• Calibration. Temperatures (choice 1.23, score 1.25, yes/no 0.85) were fitted on a held-out split that contains repeated typed-decisions cases, so they are optimistic. Expected calibration error on typed-decisions test is 0.12. Refit on your own decisions.
• Weak at inference and moderation. Paraphrase, reading comprehension, NLI and offensive-language questions are near or below a trivial baseline.
• Not a wake-word detector. It answers about a clip you hand it. It does not listen continuously.
• It picks among options. It does not extract values such as "ten minutes".
• Long states. Trained at 512 tokens. In transcript + frames mode the transcript is cut at 128 tokens, and a clip longer than 15 to 25 s (less with long questions) loses its last frames.
• Up to about 20 options. The question and options share a 192-token budget.

Licence

Apache 2.0, covering the code and the model weights.

The training data carries its own terms, and two are worth knowing before you rely on this commercially: SLURP's audio is CC BY-NC 4.0 (its text is CC BY 4.0), and AG News is distributed for research use. No training data is redistributed here.

The logo depicts a character owned by its respective rights holders and is not part of the licence grant.

Credits

Built on Ettin (JHU CLSP), the RLCD method and sequence layout of Laya (Convai Innovations, Apache-2.0), and the typed-decisions benchmark and scorer. Compared against Oscar-1, Laya, JEV-9B and AudioJev. Synthetic speech from Kokoro-82M; real speech from SLURP and MInDS-14.