You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is licensed under CC BY-NC 4.0 (see LICENSE and NOTICE): non-commercial use only. Commercial use needs a separate written license from the copyright holder. For commercial use, contact lukashcode@gmail.com or info@liriti.com. A company can also ask there for a free 30-day evaluation, to test R2Intent v1 on its own messages before buying.

R2Intent v1: intent classification with your own list of intents

Summary

R2Intent v1 picks the matching intent for a customer message from a list of intents that you give at run time. Each intent is a name with an optional short description, and you need no training examples of your intents. It is not a zero-shot model, though. It was trained on public intent data, including parts of the training splits of Banking77, HWU64, MASSIVE and CLINC150 banking and credit_cards (see Limitations), so its results on those benchmarks are in-domain results.

For benchmark comparisons that need a model trained without those training splits, the project also releases a research model, r2intent-zs-v1 (https://huggingface.co/lukashm/r2intent-zs-v1; zs means zero-shot: no training data of the test benchmarks registered in the protocol and no banking data).

R2Intent v1 was trained on English and 50 other languages. It was checked on DEV, the project's development messages, in English, Lithuanian, Polish, German and Slovak; the non-English DEV messages are machine translations. With the anyintent library it can also answer yes or no for one intent with a probability (check, a beta) and give a value for every intent (scores).

License: CC BY-NC 4.0, non-commercial use only, with a commercial license on request (see below). For commercial use, contact lukashcode@gmail.com or info@liriti.com. A company can also ask there for a free 30-day evaluation, to test R2Intent v1 on its own messages before buying.

Test results

Accuracy on the test sets of public benchmarks. R2Intent v1 was measured once, on the frozen files of this release; the other models on 2026-10-08, before this model was trained, in the project's single look at the test sets and in the baselines registered before it. The best number in each row is in bold.

Benchmark	R2Intent v1	r2intent-zs-v1	Qwen3-Embedding-8B	EmbeddingGemma 2	Laya
Banking77	0.810	0.711	0.766	0.933	0.452
HWU64	0.860	0.705	0.845	0.865	0.569
MASSIVE, 17 languages	0.774	0.587	0.832	0.660	0.311
MTOP, German, French and Spanish	0.633	0.547	0.728	0.677	0.301
xSID, Lithuanian	0.888	0.864	0.840	0.310	0.418
MInDS-14, 9 languages	0.896	0.895	0.737	0.814	0.615
SIB-200, 9 languages	0.715	0.714	0.795	0.722	0.761
Mean of the 7 benchmarks above	0.796	0.717	0.792	0.712	0.489
CLINC150 banking and credit cards	0.949	0.909	0.784	0.842	0.882
New topics (DEV, 8 topics in 5 languages)	0.781	0.779	0.641	0.546	0.482
Parameters	582 million	582 million	7.57 billion	744 million	421 million

Speed on a Kaggle T4 GPU, one message at a time with the project's measuring method: R2Intent v1 25.86 ms per message (median; 90th percentile 27.41 ms), r2intent-zs-v1 26.56 ms (median; 90th percentile 28.06 ms). The speed of the other models was not measured in this project.

R2Intent v1 was trained on public data that include the training splits of Banking77, HWU64, MASSIVE and CLINC150 banking and credit cards, with every training message close to a test message removed; it was not trained on the data of MTOP, xSID, MInDS-14 and SIB-200, although their topics are partly covered. r2intent-zs-v1 was trained without the training data of any of these benchmarks and without banking data. The training data of Qwen3-Embedding-8B and EmbeddingGemma 2 are not disclosed, and Banking77, MASSIVE and MTOP are part of the public MTEB collection, so whether those models saw them is not known. Qwen3-Embedding-8B and EmbeddingGemma 2 are shown with intent names, their better setting in these runs; Laya with its default settings. New topics are 8 DEV topics that no public dataset contains, in English, German, Lithuanian, Polish and Slovak; they were generated by the same model that generated part of this project's synthetic training data, which favours this project's models. Parameter counts are those of the published model files.

Every number of this evaluation is published, including the rows where other models score higher. The results per language, with intervals and macro-F1, and the test results of check and scores are under Test results in the full report.

Quick start

You need Python 3.9 or later, git and a Hugging Face account. On a Mac, the library needs Apple Silicon and uses the CPU. On Windows, use 64-bit x64 Python (3.12 and 3.13 are the safe choices) and, for an NVIDIA GPU, install PyTorch from the pytorch.org index before the library. The library and this model were tested on Linux, macOS and Windows, on the CPU. More details are in the library's Systems notes and its Windows page.

Install the anyintent library at the tag of this release:

pip install "git+https://github.com/lukamaga/anyintent@v0.1.0"

On this page, while logged in to Hugging Face, read the license and click Agree and request access. Hugging Face shares your user name and e-mail address with the owner of this repository. Access is granted automatically.
Create a read access token at https://huggingface.co/settings/tokens and log in with it once on this machine:

hf auth login

Or set the environment variable HF_TOKEN to the token.

Load the model and classify messages. The first run downloads the model into the Hugging Face cache: one model file of 1.19 GB and a few small files.

from anyintent import AnyIntent

clf = AnyIntent.from_pretrained("lukashm/r2intent-v1", revision="main")  # or a commit hash of this repository, to pin its files
clf.set_labels([
    {"name": "lost card", "description": "The customer reports that a bank card was lost or stolen."},
    {"name": "refund status", "description": "The customer asks when a refund will arrive."},
    "close account",  # a name alone also works
])

p = clf.classify("I can't find my card anywhere")
print(p.label, p.score, p.margin)

preds = clf.classify(["Kur yra mano pinigai?", "Chcę zamknąć konto"])  # a list in, a list out

The model reads each intent as the text name: description (the name alone when there is no description). classify returns a Prediction (a list of them for a list of messages).

label is the chosen intent. score is its score (higher is better; not a probability).
margin is how far the best intent is ahead of the second best, in score units, so a small margin means a doubtful answer. It is None when only one intent was scored.
The model runs on a CUDA GPU (fp16) or on a CPU (fp32; no CPU speed was measured). On a CPU, a few answers where two intents score almost the same can differ from those on a GPU.

Details: what pip installs, and the margin

Install the anyintent library from its repository, https://github.com/lukamaga/anyintent, at the tag pinned for this release; pip also installs what it needs (PyTorch, transformers, safetensors, huggingface_hub and numpy). The library's code has its own terms (the LICENSE file of its repository); they give no rights to the model, whose license is below.

margin is the score of the best intent minus that of the second best among the 5 intents that the retrieval step keeps (all of yours when you give 5 or fewer; see How it works).

Details: test it on your own messages

Before you rely on the model, test it on messages from your own traffic that you have labeled with the right intent. A company that runs this test for its business needs a commercial license or the free evaluation (see License and commercial use below). Save them as messages.csv with two columns named text and intent, one message per row. In the script, replace the three example intents in DESCRIPTIONS with yours, each with a short, plain description, and spell each intent exactly as in the file. The script prints the accuracy, what three margin thresholds would do and every wrong answer.

import csv
from anyintent import AnyIntent

DESCRIPTIONS = {  # every intent in your file, with a short, plain description
    "lost card": "The customer reports that a bank card was lost or stolen.",
    "refund status": "The customer asks when a refund will arrive.",
    "close account": "The customer wants to close their account.",
}

with open("messages.csv", newline="", encoding="utf-8-sig") as f:  # columns: text, intent
    rows = list(csv.DictReader(f))
assert {r["intent"] for r in rows} <= DESCRIPTIONS.keys(), "every intent in the file needs a description"

clf = AnyIntent.from_pretrained("lukashm/r2intent-v1", revision="main")
clf.set_labels([{"name": n, "description": d} for n, d in DESCRIPTIONS.items()])
preds = clf.classify([r["text"] for r in rows])  # a list in, a list out
right = [p.label == r["intent"] for p, r in zip(preds, rows)]
print(f"accuracy: {sum(right) / len(rows):.3f} on {len(rows)} messages")

# What a margin threshold would do: the answers below it go to a person
margins = sorted(p.margin for p in preds)
for share in (0.05, 0.1, 0.2):
    threshold = round(margins[int(share * len(margins))], 2)
    rest = [ok for ok, p in zip(right, preds) if p.margin >= threshold]
    print(f"threshold {threshold:.2f}: {1 - len(rest) / len(rows):.0%} doubtful, "
          f"accuracy of the rest {sum(rest) / max(len(rest), 1):.3f}")

for r, p, ok in zip(rows, preds, right):
    if not ok:
        print(f"wrong: {r['text']!r} is {r['intent']!r}, the model said {p.label!r} (margin {p.margin:.2f})")

Save the file in UTF-8: a file in another encoding can stop the script with a UnicodeDecodeError. If your spreadsheet program separates the columns with semicolons, change csv.DictReader(f) to csv.DictReader(f, delimiter=";").
On Windows, when you send the output to a file, Python can write it in the system's code page, and a letter that this code page lacks then stops the script. In cmd, with the script saved as, for example, check_messages.py, run python -X utf8 check_messages.py > out.txt. For PowerShell, see the Windows page linked above.
Each threshold line shows what share of answers would go to a person and how accurate the rest would be. Pick a threshold that suits you. Send the answers whose margin is below it to a person, or pass it to the library's HTTP server as --doubt-margin; the server then marks those answers as doubtful. The margin is in score units, not a probability. Its scale depends on the model and its settings, so choose the threshold on your own messages.
With few messages, the three lines can show the same threshold, and a line can leave no answer at or above its threshold: its accuracy of the rest then shows as zero, which means that no answer is left. The HTTP server's --doubt-margin takes only a value above zero.
Tune the descriptions and the threshold on one part of your messages and keep another part, for example one in five, for a single final test. Numbers from the messages you tuned on usually look better than the numbers on real traffic.
Leave out messages that fit none of your intents, or count them separately: the model always picks one of your intents, so on such messages it is always wrong.
Do not paste messages from messages.csv into the descriptions. That makes the test look too good (see Limitations).

How it works

One XLM-RoBERTa large network in one file (model.safetensors), fine-tuned from BAAI/bge-reranker-v2-m3, makes two passes over each message. Size: 582 million parameters, one file of 1.19 GB.
Retrieval: with a small add-on (a LoRA adapter) switched on, it turns the message and every intent text into vectors and keeps the k intents closest to the message (k = 5 in this model). An intent this step misses cannot be chosen.
Reranking: with the add-on off, it reads the message together with each of those intent texts (the cross-encoder pass) and scores each pair. It reads at most 160 tokens of the message and the intent text together; when a pair is longer, the longer of the two is cut first, so long messages and long descriptions can both be cut.
The final score is the cross-encoder score plus w times the retrieval cosine (w = 60 in this model); the highest score wins, so the model always picks one of the intents you give.
Load it with the anyintent library: it reads these settings from r2intent_meta.json and the add-on from model.safetensors; a plain transformers load uses neither.

Details: exact size

It has 581,911,553 parameters (567,755,777 in fp16; the embedding adapter's 14,155,776 in fp32), in one file of 1.19 GB (1,192,223,946 bytes).

Yes or no for one intent, and a value per intent

With the model and the intents of the quick start:

r = clf.check("I can't find my card anywhere",
              {"name": "lost card", "description": "The customer reports that a bank card was lost or stolen."})
print(r.yes, r.probability)  # yes or no for this one intent, and the probability of a match (check is a beta, see below)

s = clf.scores("I can't find my card anywhere")  # every intent of set_labels
print(s.label, s.values["lost card"])  # the chosen intent, and the value of one intent

check answers yes or no for one intent, with the probability that the message expresses it when a match and a non-match are taken as equally likely before the message is read (the table below shows what changes when the intent is rare). scores gives every intent of your list a value: for an intent the model read with the message (How it works), the chance that it is the right one; an intent it did not read gets 0.0. Both were fitted on one part of DEV and checked on the rest (held out), for intents written with a description and on lists of 4 to 63 intents.

Beta: on the 19 English DEV sets of its clause C4, the probabilities of check were off by 0.0566 [0.0375, 0.0790] on average, above the pre-registered limit of 0.050 (all other clauses passed). On DEV sets in domains close to its training data (sets on the seen-domain list; for information, not a clause of the rule), they were off by 0.066 [0.043, 0.092]. Treat the probability of check as approximate for any intent. It was kept in v1 as a beta after this result.

If the intent is rare in your messages, a yes from check is right less often (held-out DEV, with 95% intervals):

messages that express the intent	yes answers that are right [95% interval]
1 in 2	0.888 [0.862, 0.909]
1 in 5	0.665 [0.610, 0.715]
1 in 20	0.295 [0.248, 0.346]

Details: what check and scores return, and every held-out DEV result

Besides classify, the anyintent library gives this model two calibrated outputs: check (beta), yes or no for one intent with a probability, and scores, a value for every intent of your list (both are in the code above). Each was fitted on one part of DEV and checked on the rest (held out). scores passed its release rule there; check did not meet one clause of its rule and ships as a beta (above). Every number below is a held-out DEV result with its 95% interval. The calibration error (ECE) is the average gap between the probability and how often the intent is right; lower is better. Its interval comes from resamples, on which ECE tends to run higher, so the interval can lie above the value.

check (beta): yes or no for one intent. check(text, intent) reads the message together with one intent (a name and a description, as in set_labels) and returns a CheckResult with yes and probability. The probability is the chance that the message expresses the intent, assuming that a match and a non-match were equally likely before the message was read. yes means a probability of 0.5 or more (the default threshold, the only one measured). The answer for one intent does not depend on your other intents, and check does not change the intents of set_labels.

Calibration error (ECE): 0.014 [0.008, 0.03001]. Balanced accuracy at 0.5: 0.873 [0.856, 0.888], the mean of the right yes rate on the right intent, 0.853 [0.832, 0.873], and the right no rate on other intents of the same list, 0.893 [0.863, 0.915]. On intents close to the right one (the other intents among the 5 that classify reads with the message), it wrongly says yes for 0.420 [0.391, 0.449] of them. So a yes does not mean that the intent is the best one; classify picks one.

The 19 English DEV sets of clause C4 (Beta above), which the library's documentation calls the unseen English DEV sets, are HINT3, the project's generated sets and the fresh V-37 and V-IND sets (generated sets of domains and industries held out of training), a wider group than EN_unseen under Results at a glance.

scores: a value per intent. scores(text) works on the intents of set_labels and returns a ScoresResult with:

values: a value from zero to one for every intent. For a read intent it is the chance that it is the right intent; an intent not read gets 0.0, which means not read, not impossible.
remainder: the chance that the right intent is one of your intents that the model did not read. The model reads 5 intents with each message (How it works), so with 5 intents or fewer every intent is read and the remainder is zero.
read: the intents the model read with the message.
label: the intent with the largest value, which is the label classify gives.

The values and the remainder add up to one: they assume that one of your intents is right, so the remainder is not a 'none of these' answer (there is no such answer).

Calibration error (ECE) of the largest value 0.019 [0.012, 0.035], of every read value 0.005 [0.004, 0.010] and of the remainder 0.006 [0.005, 0.018]. The ECE of the remainder by the number of intents in the list: 6 to 20 intents 0.008 [0.004, 0.034]; 21 to 40 intents 0.012 [0.006, 0.024]; 41 or more intents 0.013 [0.011, 0.035]. The right intent was among the 5 read intents for 0.928 [0.912, 0.942] of the messages (recall@5).

For information (no clause of the rule), on held-out DEV rows the largest value understates the chance that the intent is right for messages with an e-mail wrapper (ECE 0.054 [0.037, 0.077]) and messages without diacritics (ECE 0.047 [0.0297, 0.068]). It overstates it for intents given by their names only (ECE 0.041 [0.027, 0.070]). On VAL, rows held out from the training datasets (Results at a glance), the ECE of the largest value was up to 0.082 [0.058, 0.116] (Banking77, where it understates the chance) and up to 0.059 [0.033, 0.096] (MASSIVE hu-HU, where it overstates it).

Both are calibrated for intents written with a description (check, a beta, missed its calibration limit on one group of DEV sets: see Beta above). The calibration holds at this model's settings (w = 60, k = 5, label texts name: description) and with the library's engines auto and eager on a GPU, where it was checked. An intent given by its name alone was not part of the release rule (the library warns). Both were measured on lists of 4 to 63 intents; above 63 the remainder is extrapolated. The full report (Calibrated outputs check and scores) gives the rule, every clause and every result, and Test results in the full report holds their test information.

Run it as a local service (HTTP or MCP)

The anyintent library also has an HTTP server, and an MCP server for AI agents and editors that speak the Model Context Protocol. Both are optional extras that only call the library: they load this model once and run on your machine. Install the one you need (the MCP server needs Python 3.10 or later):

pip install "anyintent[server] @ git+https://github.com/lukamaga/anyintent@v0.1.0"
pip install "anyintent[mcp] @ git+https://github.com/lukamaga/anyintent@v0.1.0"

Then start it with this model (add --revision with a tag or a commit of this repository to pin its files):

anyintent-serve --model lukashm/r2intent-v1 --lists-dir ./intent-lists
anyintent-mcp --model lukashm/r2intent-v1 --lists-dir ~/.anyintent/lists

Your messages stay on your machine, and their texts are not logged unless you start a server with --log-texts. The library and the servers do not mark check as a beta in their answers.

Details: routes, tools and options

The HTTP server listens on this machine only unless you give it another --host, and answers /classify, /check (yes or no for one intent; a beta, see above), /scores (a value per intent) and saved intent lists under /lists. The MCP server speaks MCP over stdio and offers the tools classify_intent, check_intent (a beta), score_intents and tools for saved intent lists. The library's docs/server.md (https://github.com/lukamaga/anyintent/blob/v0.1.0/docs/server.md) gives every option, the API key and the limits of the HTTP server, and its docs/mcp.md (https://github.com/lukamaga/anyintent/blob/v0.1.0/docs/mcp.md) the options, the tools and a client configuration of the MCP server.

Intended use

Routing and classifying short customer messages into your own list of intents, for example in a support chat or a ticket queue (for e-mails, see Limitations).
A first triage step before a person or a larger model. The model always picks one of your intents, so pass answers with a small margin (doubtful answers) to that person or model, using a threshold chosen on your own labeled messages.

Out-of-scope use

Benchmark comparisons that need a model that never saw the benchmark: this is not a zero-shot model. Use r2intent-zs-v1, the research model of this project, for those; its test numbers come with notes about its training rows that are close to test messages.
Decisions about people (for example about credit, insurance, jobs or access to a service) without review by a person.
Long documents: the model was not tested on them, and its cross-encoder reads at most 160 tokens of message and intent text together.
Commercial use without a separate license (see License and commercial use below).

Languages

Training: English and 50 other languages (listed as language tags at the top of this page; these are training languages, not languages the model was checked in).
Checked on DEV: English, and machine translations of English DEV messages into Lithuanian, Polish and Slovak (OPUS-MT) and German (EuroLLM-9B-Instruct-2512). Slovak is not in the training data.
Lithuanian has no human-written training rows: all its training text is machine-translated or generated.

Details: training rows by language, and VAL

Human-written training rows in 49 languages besides English, from MASSIVE, Multi3NLU++ and PRESTO; with en-US, all 51 locales of MASSIVE 1.0 are in the training data; ca-ES, which MASSIVE 1.1 adds, is not.
Machine-translated training rows in German, Spanish, French, Italian, Lithuanian, Latvian, Polish and Russian, and rows written directly in Lithuanian and Polish by EuroLLM-9B-Instruct-2512.
German and Slovak have no accuracy group of their own: the full report's release gate (NW-P) pools them as de+sk_unseen, and its calibration tables and Appendix B also give German and Slovak numbers.
VAL (below) includes the MASSIVE dev split, human-written, in 18 locales (da-DK, de-DE, el-GR, en-US, es-ES, fi-FI, fr-FR, hu-HU, it-IT, lv-LV, nb-NO, nl-NL, pl-PL, pt-PT, ro-RO, ru-RU, sl-SL and sv-SE).
No other language has a DEV number in this card.

Results at a glance (DEV and VAL only)

The test-set numbers are under Test results above; the numbers in this section are from DEV and VAL only. In this section, Z is r2intent-zs-v1, the research model of this project.

Accuracy is the share of messages whose intent the model picks correctly, written as a fraction rather than a percentage. Numbers next to a limit of the protocol's rules, or next to zero, carry extra digits (for example 0.03001), so that rounding never moves them onto or across that limit. DEV is the set of development messages the project used to choose and check models; VAL is a set of rows held out from the public training datasets.

Against Z on DEV

group	this model	Z	this model minus Z [95% interval]	margin allowed below Z
English DEV (EN_DEC)	0.842	0.842	-0.0001 [-0.0151, 0.0153]	0.010
Lithuanian and Polish DEV (LTPL)	0.753	0.748	0.004 [-0.012, 0.021]	0.010
English DEV, sets outside the seen-domain list (EN_unseen)	0.819	0.810	0.009 [-0.006, 0.024]	0.010
Lithuanian and Polish DEV, sets outside the seen-domain list (LTPL_unseen)	0.753	0.756	-0.004 [-0.021, 0.013]	0.010
DSTC11 banking and finance (GUARD)	0.926	0.940	-0.014 [-0.042, 0.008]	0.020

This model met every margin of its release gate against Z (NW-P), and each 95% interval includes zero: on these DEV messages the two models show no clear difference (the intervals cover the choice of messages only, not the variation between training runs).

Details: the release gate and its DEV groups

Before release, the project's protocol required this model to be no more than a fixed margin below Z on every group of its release gate against Z, NW-P (the full report gives all its groups). Both models were scored in the same job on the same DEV messages. The groups in this table:

domain_seen_by_P (the seen-domain list): a list of DEV sets in domains close to this model's training data, fixed by a rule before the training data were built (the full report names them).
English DEV (EN_DEC): the human-written sets (HINT3 and SGD) and the synthetic sets (Bitext and the project's generated sets) each make up half of the score. DSTC11 banking, DSTC11 finance and DSTC11 insurance are left out because they are in the domains of the public training data, and CLINC150's English set is not in this group.
Lithuanian and Polish DEV (LTPL): machine translations into Lithuanian and Polish of the English DEV sets of CLINC150 and the project's generated sets; CLINC150 is on the seen-domain list.
Sets outside the seen-domain list (EN_unseen, LTPL_unseen): the English human-written and synthetic sets that are not on the list (HINT3 and the project's generated sets), and their Lithuanian and Polish copies (the project's generated sets); every message counts the same. The rule is approximate: HINT3's DEV set stays in these groups, although HINT3's training split is in this model's training data.
DSTC11 banking and finance (GUARD): English customer-service messages of DSTC11 banking and DSTC11 finance; DSTC11 is not in this model's training data; the protocol allows a wider margin on this group.

The intervals cover the choice of messages only: this model was trained once and Z is one run of a recipe trained with three seeds, so the variation between training runs is not in them.

Changed messages (DEV)

change	what it does to the message	accuracy on the changed messages, this model minus Z [95% interval]	accuracy drop from the change, Z's drop minus this model's [95% interval]	in plain words
no diacritics	Lithuanian and Polish messages written without accented letters	0.037 [0.025, 0.048]	0.035 [0.024, 0.045]	drops less than Z
one typo	one typing error in one word	0.009 [0.004, 0.0154]	0.006 [0.002, 0.011]	drops less than Z
e-mail wrapper	a fixed e-mail greeting and signature around the message	0.091 [0.081, 0.102]	0.088 [0.077, 0.099]	drops less than Z
label names only	the intents are given by their names only, without descriptions	0.006 [-0.001, 0.013]	0.002 [-0.006, 0.011]	no clear difference (the interval includes zero)

By the 95% interval of the accuracy drop, this model drops less than Z with no diacritics, one typo and an e-mail wrapper; it shows no clear difference with label names only; this comparison is not held out.

Details: how the messages were changed, and the caveats

These rows are the DEV messages of the project's earlier noise check of Z (the v1.1 probe in the full report): the English DEV sets of Bitext, DSTC11 insurance, HINT3, SGD and the project's generated sets, and the Lithuanian and Polish copies of the DEV sets of CLINC150 and the project's generated sets. They are not exactly the groups above. Each message was scored unchanged and with one change made by a fixed rule: no diacritics on the Lithuanian and Polish rows; each other change on the English and on the Lithuanian and Polish rows, counted half each. Each number is this model minus Z with its 95% interval: first the accuracy on the changed messages, then the accuracy drop from the change, as Z's drop minus this model's (positive: this model drops less).

This model was also trained with these changes: no diacritics, one typo, e-mail wrapper and label names only (Training in the full report). Z differs from it in training data too. Z was itself trained with diacritic stripping on 0.15 of its Lithuanian and Polish groups (cross-encoder only; this model: 0.3, and also in its retrieval adapter). So these rows compare the two models and do not isolate what that training adds. The e-mail greetings and signatures of this check were not used in training.

The intervals cover the choice of messages only: this model was trained once and Z is one run of a recipe trained with three seeds; the variation between training runs under a change was not measured, so a small difference is not evidence either way. In the full report, the wording 'better on noisy text' means, as the protocol registered it, that the point estimate of this model minus Z in accuracy on the changed messages is above zero; the last column here goes by the 95% interval of the drop instead. Some of these DEV sets are in the domains of this model's training data, HINT3's training split is in that data, and the Lithuanian and Polish rows include copies of CLINC150's DEV set, whose English original was used to choose this model's checkpoint. So this comparison is not held out. Section (10b) of the full report gives every caveat of this re-measure.

VAL: held-out rows of the training datasets

Accuracy on the VAL rows. VAL rows are held out from the same public datasets this model was trained on, so they are close to its training rows and can overstate what the test sets will show.

VAL set	intents scored	this model	Z	this model minus Z [95% interval]
Banking77 (held-out part of its training split)	75	0.811	0.690	0.120 [0.087, 0.151]
HWU64 (held-out part of its training split)	62	0.857	0.701	0.156 [0.099, 0.214]
CLINC150 banking and credit_cards (its validation split)	29	0.969	0.938	0.031 [0.009, 0.059]
MASSIVE (its dev split, 18 locales)	57	0.802	0.615	0.187 [0.169, 0.202]

Details: what the VAL table compares

Z was trained on none of these training splits (for CLINC150, on none of its banking and credit_cards rows), so the gap shows mainly what the in-domain training data adds on rows close to that data, though the two models also differ in their other training data. Each set is scored only against its intents that have training rows (the column 'intents scored'); the intents without any training row are left out with their rows.

Speed

On a Kaggle T4 GPU, measured with the protocol's method 3b (fp16, torch.compile with CUDA graphs, intent vectors computed in advance), one CLINC-length DEV message at a time with k = 5: median 25.86 ms per message, 90th percentile 27.41 ms (section (10d) of the full report). The quick start's settings do not compile the model (the anyintent default is compile=False) and were not measured. No CPU speed was measured.

Limitations

The model always picks one of the intents you give: it has no 'none of these fits' answer, so a message that fits none of them still gets one. A small margin marks a doubtful answer, which you can route elsewhere with a threshold of your own; the margin compares the best two of the intents it scored, so it does not show whether either fits. This card reports no measurement on messages that fit none of the given intents. check answers about one intent at a time, and the remainder of scores is the chance that the right intent is one the model did not read; neither is a 'none of these' answer.
check is a beta: it missed its calibration limit on one group of DEV sets, so treat its probability as approximate for any intent (Beta, under 'Yes or no for one intent, and a value per intent').
Lithuanian and Polish DEV numbers are measured on machine translations of English messages, not on messages written by Lithuanian and Polish customers.
Most English training messages are machine-made. Of the 155,722 English training rows, 37,636 were generated by Qwen2.5-14B-Instruct and Qwen2.5-7B-Instruct, and 80,063 more come from Bitext's datasets, which Bitext generated with its own NLG tools: about 76% together. These counts come from the training data files, before the trainer removed some English rows and capped the rows per intent. The Lithuanian and Polish training text is mostly machine-translated, or generated by EuroLLM-9B-Instruct-2512.
It was trained once, with a single training seed, so this card does not show how much the numbers move between training runs.
The e-mail wrappers used in training put a greeting and a closing around a message; real e-mails with quoted threads, disclaimers or long bodies are not covered.
The margin is not a calibrated probability: it is in score units (the cross-encoder score plus w times the cosine), its scale depends on the settings, and a threshold chosen for one model or setting does not carry over to another. For a calibrated value per intent, use scores (above).
Write intent descriptions like those in the quick start: a name and a short, plain description of what the customer wants. Do not paste example messages from your own test data into the descriptions: that makes the test numbers look better than the numbers on real traffic.
The model was trained on only part of the training splits of Banking77, HWU64, CLINC150 banking and credit_cards and MASSIVE. After the removal of training rows close to test messages (rule E4 of the protocol) and the other data rules, about 23% of the Banking77 and 33% of the HWU64 training rows that were not held out for VAL remain. So do about 53% of the CLINC150 banking and credit_cards training rows and about 12% of the MASSIVE en-US utterances (the full report, Disclosures, 'What survives E4').

Details: check and scores in numbers

On the 19 English DEV sets of its clause C4, the probabilities of check were off by 0.0566 on average, above the pre-registered limit of 0.050. On DEV sets in domains close to its training data they were off by 0.066 (for information, not a clause of the rule). The decision to ship it with its parameters as fitted was taken after that result (see 'Yes or no for one intent, and a value per intent' above and 'Calibrated outputs check and scores' in the full report).

The probabilities of scores hold for intents written with a description, at this model's settings and with the library's engines auto and eager on a GPU, as measured on DEV (Yes or no for one intent, and a value per intent, above). For messages with an e-mail wrapper and messages without diacritics the largest value of scores understates the chance that the intent is right, and for intents given by their names only it overstates it (numbers above).

License and commercial use

CC BY-NC 4.0 (see LICENSE): you may use, share and adapt this model for non-commercial purposes, with attribution. Commercial use needs a separate written license from the copyright holder. For commercial use, contact lukashcode@gmail.com or info@liriti.com. A company can also ask there for a free 30-day evaluation, to test R2Intent v1 on its own messages before buying. The base models' Apache-2.0 and MIT licenses and the third-party notices in NOTICE still apply to their parts: when you share this model or a model made from it, share LICENSE, NOTICE, LICENSE-APACHE-2.0.txt, LICENSE-MIT-FlagEmbedding.txt, LICENSE-MIT-fairseq.txt and hint3_alterations.json with it. Third-party notices (training data, base models and the translation and generation models): NOTICE. Copyright © 2026 Lukaš Patrik Magalinski.

Access: when you request access, Hugging Face shares your user name and e-mail address with the owner of this repository. They are used only to manage access and to answer license requests.

Contact

Questions, feedback and collaboration: lukashcode@gmail.com or info@liriti.com. LinkedIn: https://www.linkedin.com/in/lukashm/

Citation

@misc{magalinski2026r2intent,
  author       = {Lukaš Patrik Magalinski},
  title        = {R2Intent v1: intent classification with your own list of intents},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/lukashm/r2intent-v1}},
  url          = {https://huggingface.co/lukashm/r2intent-v1}
}

Full report (PROTOCOL)

In this report P is R2Intent v1 and Z is r2intent-zs-v1 (model v1.1 of the protocol).

Everything below is the full report that the project's registered protocol (PROTOCOL) requires for this release, each statement with the record it comes from. A reference such as 'PROTOCOL 2026-10-13 00:06' names the protocol entry logged at that date and time; the protocol is published in the repository of the anyintent library, https://github.com/lukamaga/anyintent.

Track T3, public model: trained on public data including the training splits of Banking77, HWU64, MASSIVE and CLINC150 banking and credit_cards. This is the track's label as PROTOCOL E3 registered it; the model was trained on parts of those splits (Disclosures, 'What survives E4'). This model is not a zero-shot model and is never described as zero-shot; the project's zero-shot results come from the separate model Z (track T1; model v1.1 of the protocol), released next to it under its own label as r2intent-zs-v1.

What this model is

One XLM-RoBERTa large backbone in one file (model.safetensors: backbone and head in fp16, the embedding adapter under r2intent_embed_lora.* in fp32), used in two passes: an embedding pass with the LoRA adapter on (retrieval of the top-k labels by cosine) and a cross-encoder pass with the adapter off (one logit per message and label text pair); final score = cross logit + w x cosine.

Settings: w = 60.0, k = 5, no cascade (tau null), label texts v2 (name: description). Adapter: rank 32, alpha 64.0.

Inference: label texts are given at inference; no few-shot example block is packaged or used, and no P with examples at inference exists. Load it with the anyintent library (AnyIntent.from_pretrained(<this folder or repository>)); the library reads r2intent_meta.json and the adapter under its R2Intent names.

Shipped model: p_full (P_full); ladder step 1: P_full passed NW-P of the gate and repair ladder of PROTOCOL E5 (9) (record final_decision_20261011T191403Z.json, sha256 f7297b34c6a5294a4e550ce40df06ac8088fb97593123da008498b57b286526f). The ladder ships a finished public model unless even the 0.05 mixes of its steps 6 and 7 fail its gate; steps 4, 5 and 7 ship a model without any lever, and step 6 a mix in which Z holds at least 0.7 of the weights.

Levers of this model: the D pack, V-37, V-IND, e-mail wrappers (V-E), V-TYPO, V-MIX, V-ADIA (noisy messages in the retrieval adapter).

The gate NW-P against the zero-shot model Z (DEV only)

NW-P compares this model with Z (configs.B: the v1.1 ONE-0 at w 60, k 5, tau null), both re-scored in the same job, on the decision rows of P's DEV (the data 1.2 DEV row for row and its ML copies); point estimates, compared unrounded; margins 0.010 on EN_DEC, LTPL, EN_unseen and LTPL_unseen, 0.015 on lt_unseen, pl_unseen and de+sk_unseen, 0.020 on GUARD. It passes when every clause holds. Result: PASS.

quantity	this model	Z	difference	margin	holds
EN_DEC (0.5 REAL + 0.5 PIPE, without the pub-domain pools)	0.842	0.842	-0.0001	-0.010	yes
LTPL	0.753	0.748	0.004	-0.010	yes
EN_unseen	0.819	0.810	0.009	-0.010	yes
LTPL_unseen	0.753	0.756	-0.004	-0.010	yes
lt_unseen	0.749	0.752	-0.004	-0.015	yes
pl_unseen	0.757	0.761	-0.004	-0.015	yes
de+sk_unseen	0.772	0.770	0.002	-0.015	yes
GUARD (DSTC11 banking and finance, a tripwire)	0.926	0.940	-0.014	-0.020	yes

Pools of at least 100 decision rows more than 0.05 below Z: none.

Rule B (PROTOCOL 2026-10-11 22:01): while P0 passes NW-P against Z, a model built from P_full ships only if it also passes NW-P against P0, that is, it is at least P0 minus the same margins on every clause; here P0 did not pass NW-P against Z, so rule B did not bind and P_full ships on its NW-P against Z (for information, it also passes NW-P against P0, rule B record).

domain_seen_by_P (one list for every P model, fixed before data pub was built): abcd_dev, bitext_dev, clinc_dev, dstc11_banking_guard, dstc11_finance_guard, dstc11_insurance, mixed_dev, nlupp_hotels_dev, sgd_unseen, with every language copy. These pools are left out of the unseen groups.

Resolution of NW-P (simulation on v1.1 DEV stand-ins, E2): a harmless P fails by chance about 7% (row model) to 14% (cluster model) of the time when it differs from Z as much as A1 from ONE-0, and 16% to 25% (row model) when it differs 1.5 to 2 times as much, the more likely case for P. A hidden loss on the unseen pools of 1.5 points is caught about 94 to 98% of the time, 1 point about 72 to 84%, 0.5 point about 33 to 52%; no rule on this DEV sees a loss below about 0.5 point.

Recomputed before data pub was built (PROTOCOL note of 2026-10-09 12:06 under E2, DRAWS record sha256 25e1c9d1b7ec620679b7e71a273c1411989c928c4154c862b32c3c2feba4c63f; E2's method with Z's seeds 0, 1 and 2 as the harmless pair, margins unchanged; minimum to maximum over the seed pairs): a harmless P fails by chance 0.077 to 0.101 of the time (row model) when it differs from Z as much as A1 from ONE-0, 0.174 to 0.214 when it differs 1.5 times as much and 0.268 to 0.303 when it differs twice as much, and 0.147 to 0.213 under the cluster model; observed: all 6 ordered pairs of Z's seeds pass NW-P. A model equal to Z fails NW-P by chance with a probability of roughly 0.1 to 0.2 under these models; the gate and repair ladder of PROTOCOL E5 (9) is the registered answer to that. The list domain_seen_by_P removes 1,559 decision rows from EN_unseen (1,924 of 3,483 remain) and so lowers its power (PROTOCOL E5 (7)).

Reports of PROTOCOL E5 (10) (DEV; information only: they never make a model ship or stop one)

(10a) Noisy and clean targets against Z

Paired differences with 95% cluster intervals (2,000 resamples, seed 0, clusters base pool and gold label; VAL: gold label).

target	this model minus Z [95% CI]	wording (registered: point estimate above 0 on noisy text)
e-mail wrapper (EN_DEC and LTPL rows, 0.5 / 0.5)	0.095 [0.069, 0.125]	better on noisy text
one typo (seeds 0, 1, 2; 0.5 / 0.5)	0.002 [-0.008, 0.014]	better on noisy text
label names only (0.5 / 0.5)	0.004 [-0.007, 0.0152]	better on noisy text
no diacritics (LTPL rows)	0.046 [0.028, 0.065]	better on noisy text
clean EN_DEC	-0.0001 [-0.0151, 0.0153]	within 0.010 of Z on clean text
clean LTPL	0.004 [-0.012, 0.021]	within 0.010 of Z on clean text
clean target (0.5 EN_DEC + 0.5 LTPL)	0.002 [-0.009, 0.013]
V-37 fresh pools (English)	0.013 [-0.002, 0.030]
V-IND fresh industries (0.5 EN_IND + 0.5 LTPL_IND)	0.029 [0.0146, 0.043]
V-IND fresh copies with no diacritics (LTPL_IND rows)	0.079 [0.055, 0.104]
abcd_dev (information)	-0.012 [-0.034, 0.009]
bitext_dev telco and retail ecommerce rows (information)	0.029 [0.000, 0.071]
ind_dev_seen_industry (favours P_full by construction)	0.007 [0.000, 0.0203]

What the levers add together (P_full minus P0): en_dec -0.002 [-0.018, 0.011]; ltpl 0.009 [-0.003, 0.022]; clean 0.003 [-0.007, 0.013]; email 0.130 [0.104, 0.160]; typo1 0.006 [-0.001, 0.014]; names 0.011 [-0.0004, 0.021]; nodia 0.046 [0.032, 0.062]; v37 0.019 [0.005, 0.035]; vind 0.024 [0.011, 0.037]; vind_nodia_fresh_copies 0.072 [0.053, 0.092]; abcd_dev -0.005 [-0.022, 0.0105]; bitext_dev_telco_retail 0.021 [-0.034, 0.079]; ind_dev_seen_industry 0.013 [0.000, 0.034].

VAL (the carve-outs of E2 (5), information; this model minus Z [95% CI], clusters gold label; without the 8 closed-inventory labels that have no training row, whose 3,061 VAL rows were not scored): VAL_mean 0.124 [0.106, 0.142]; banking77_train 0.120 [0.087, 0.151]; clinc_bank_validation 0.031 [0.009, 0.059]; hwu64_train 0.156 [0.099, 0.214]; massive_dev 0.187 [0.169, 0.202].

(10b) Noise re-measure on the rows of the v1.1 probe

Paired differences on the decision rows of the v1.1 probe with 95% cluster intervals (2,000 resamples, seed 0, clusters (base pool, English source row); noise_compare.json rules). N(c) is the accuracy under the change c (nodia: LTPL_ALL; typo1, email and names: 0.5 EN_NONCLINC + 0.5 LTPL_ALL), the clean rows are the same rows unchanged, a model's loss under c is N(c) minus its clean accuracy, and the difference of losses is the first model's loss minus the second's.

Within 0.010 of Z on clean text: yes (clean rows, this model minus Z: EN_NONCLINC 0.004 [-0.003, 0.012], LTPL_ALL 0.002 [-0.009, 0.013]); no changed input clearly worse (at least -0.020 on each of the seven noisy parts): yes.

change	N(c): this model minus Z [95% CI]	clean rows [95% CI]	difference of losses [95% CI]	wording (registered: N(c) point estimate above 0)
nodia (no diacritics; N = LTPL_ALL)	0.037 [0.025, 0.048]	0.002 [-0.009, 0.013]	0.035 [0.024, 0.045]	better on noisy text
typo1 (one typo; seeds 0, 1, 2)	0.009 [0.004, 0.0154]	0.003 [-0.004, 0.011]	0.006 [0.002, 0.011]	better on noisy text
email (e-mail wrapper)	0.091 [0.081, 0.102]	0.003 [-0.004, 0.011]	0.088 [0.077, 0.099]	better on noisy text
names (label names only)	0.006 [-0.001, 0.013]	0.003 [-0.004, 0.011]	0.002 [-0.006, 0.011]	better on noisy text

Groups of at least 1,000 decision rows more than 0.020 below Z under a change: REAL4 under caps (-0.0202, 1,680 rows).

What the levers add together on the probe rows (P_full minus P0, a binding comparison of noise_compare.json):

change	N(c): P_full minus P0 [95% CI]	clean rows [95% CI]	difference of losses [95% CI]
nodia (no diacritics; N = LTPL_ALL)	0.039 [0.028, 0.05001]	0.005 [-0.005, 0.0147]	0.034 [0.023, 0.044]
typo1 (one typo; seeds 0, 1, 2)	0.013 [0.007, 0.018]	0.008 [0.001, 0.014]	0.005 [0.001, 0.009]
email (e-mail wrapper)	0.121 [0.110, 0.132]	0.008 [0.001, 0.014]	0.113 [0.102, 0.124]
names (label names only)	0.009 [0.003, 0.016]	0.008 [0.001, 0.014]	0.002 [-0.006, 0.0098]

Clean rows, P_full minus P0: EN_NONCLINC 0.011 [0.003, 0.019]; LTPL_ALL 0.005 [-0.005, 0.0147].

Caveats of the re-measure (noise_compare.json, word for word, except that the name of an internal code module is replaced by a description):

clinc_dev, bitext_dev, dstc11_insurance and sgd_unseen are in domain_seen_by_P, HINT3 is in P's data and clinc_dev is P's selection pool, so the comparison with Z on these rows is not held out; the difference of losses is the readable robustness quantity.
The run-to-run standard error of a difference on these rows was estimated only in a working analysis on clean rows (E5 (15)); noisy parts have larger, unmeasured standard errors, so a small difference under a change is not evidence either way.
The lt and pl copies of clinc_dev stay in LTPL_ALL (P's checkpoint selection used English clinc_dev only).
DEV only: the lt, pl, de and sk rows are machine translations of English CLINC and synthetic messages, and the changes are fixed rules (the project's noise module, unchanged), not real user text; one training seed per model.
This is a report per noisy target; it does not decide shipping (E5 (9) and (10)).

(10c) V-V3S (label texts v3s-A at inference, no training)

Eligible, not applied: it did not pass VR against the shipped model (V-V3S is not kept: the shipped model keeps its v2 label texts). Eligible by the D18 rule with the D23 recount of the six L1 DEV sets, then judged by VR against the shipped model on DEV and VAL; VR (c) reads its 95% label-cluster lower bound as the 2.5th percentile, the stricter reading (PROTOCOL 2026-10-12 13:07). VR result (PROTOCOL 2026-10-12 13:07; record sha256 13fba8d155424f1169e57b16144f2f0bf443097e46fe80d47d26c20dfeb8d407; V is this model with the v3s-A label texts, M0 this model with its v2 texts; failed clauses: a, b:en_dec, b:ltpl, b:en_unseen, b:ltpl_unseen, b:val_mean, b:lt_unseen, b:pl_unseen, b:desk_unseen, b:val:banking77_train, c): (a) V does not pass NW-P against Z (V against Z: EN_DEC 0.826 against 0.842, fails; LTPL 0.715 against 0.748, fails; EN_unseen 0.778 against 0.810, fails; LTPL_unseen 0.702 against 0.756, fails; lt_unseen 0.695 against 0.752, fails; pl_unseen 0.708 against 0.761, fails; de+sk_unseen 0.721 against 0.770, fails; GUARD 0.907 against 0.940, fails); (b) V minus M0 against its margin, with the one-sided 95% label-cluster lower bound where the rule has one: EN_DEC -0.016 (margin 0.010, bound -0.038), fails; LTPL -0.037 (margin 0.010, bound -0.055), fails; EN_unseen -0.042 (margin 0.010, bound -0.063), fails; LTPL_unseen -0.051 (margin 0.010, bound -0.070), fails; VAL_mean -0.018 (margin 0.010, bound -0.029), fails; lt_unseen -0.053 (margin 0.015), fails; pl_unseen -0.049 (margin 0.015), fails; de+sk_unseen -0.051 (margin 0.015), fails; GUARD -0.019 (margin 0.020), holds; VAL banking77_train -0.053 (margin 0.020), fails; VAL clinc_bank_validation -0.002 (margin 0.020), holds; VAL hwu64_train -0.008 (margin 0.020), holds; VAL massive_dev -0.0097 (margin 0.020), holds; (c) EN_DEC gain, V minus M0, -0.016 with the 95% label-cluster lower bound -0.042, against at least +0.010 with a bound above 0: fails; (d) every quantity exists: yes.

(10d) Latency on a Kaggle T4 (K2)

K2 measured this model file (model.safetensors sha256 ba8d2e74f973a079d15581386ffdf2697307607078e8a3e26b10cf0490919c26) in the release frozen before it; this card was then built again with the K2 record (PROTOCOL E5 (10d)). p50 single message 25.86 ms, p90 27.41 ms (Kaggle T4, method 3b, at k_final 5, CLINC-length DEV messages; the library's default engine, with torch.compile, which the anyintent library runs only with compile=True); per language: en 26.33 ms, lt 27.82 ms, pl 28.83 ms.

Calibrated outputs check and scores (PROTOCOL 2026-10-13 00:06)

check (yes or no for one intent, with a probability) and scores (a value from zero to one for every intent the model reads with the message, and the remainder: the chance that the right intent is one it did not read) are two outputs of the anyintent library with this model. Each was fitted on the calibration part of DEV and checked once on the held-out part by the ship rule of PROTOCOL 2026-10-13 00:06 ('two calibrated outputs of the anyintent library for R2Intent v1, check (yes or no for one intent, with a probability) and scores (a value from 0 to 1 for each intent the model reads with the message, and the chance that the right intent is one it did not read)'), registered before any calibration number existed. Under that rule a function that fails a clause would not be in v1 and the library would refuse it. After the result, an exception to that rule was made for check, which ships as a beta (Outcome, below). P's registered numbers were computed without them. Record newfx_calibration_20261013T080704Z.json (sha256 a7ebb3ff746cc9ac89925d192a1ac9cfd2b943235f8641b6df2c331b5a55e7b0; status FINAL; run 20261013T080704Z). Beta record newfx_beta_20261013T141551Z.json (sha256 7476fa3cbaf0f7b5dcf9c87996c71d13cb5783ff1a9dc493090a37ab07db3801; status FINAL; run 20261013T141551Z) of tools/newfx_build/newfx_beta.py: it names this record and changes none of its numbers, and the calibration file names it (dev_record).

Outcome: check ships as a beta: it failed C4 of the registered rule, so under that rule it would not be in v1; it ships with its parameters exactly as fitted, a course chosen after this result and logged in PROTOCOL 2026-10-13 12:07 ('two notes on check and scores in v1'), and every clause and number below is as measured; scores ships. Calibration file r2intent_calibration.json (sha256 bae5dcdb89ffaf51aad19d396d20460241f47915b32c64af711865e176538285) at the release root, with the parameters of check and scores (check as a beta: the file lists it under the key beta and names the beta record in dev_record).

Split: 20,486 DEV rows of 644 intents (clusters: base pool and gold label) in 85 pools; the calibration part 7,081 rows of 223 intents, the held-out part 13,405 rows of 421 intents (Appendix B, information, gives every count). Here EN_unseen and LTPL_unseen are the groups of PROTOCOL 2026-10-13 00:06 (3): 19 English DEV sets (the release gate's en_unseen sets plus the fresh V-37 and V-IND sets) and 26 Lithuanian and Polish copies of them, wider than the NW-P groups of the same names. Intervals: 2,000 resamples of the held-out clusters (seed 0); scores one-way, clusters (base pool, gold label) of the held-out rows; check two-way: a negative pair weighted by its message cluster's and its negative intent's draw counts. Every ECE uses 15 equal-width bins. The interval of an ECE comes from resamples, in which an ECE runs higher, so it can lie above the value.

Fit of scores (calibration part): T 2.406 (1/T, 0.416 with standard error 0.007), gamma -1.502 (0.335), eta 0.104 (0.015), kappa 0.801 (0.092); converged yes after 8 Newton iterations; 7,081 calibration rows, 7,011 of lists over 5 intents, of which 6,540 with the right intent read and 471 not read (at least 20 of each needed). Fit of check (calibration pairs): a 0.274 (standard error 0.006), b -7.892 (0.183); converged yes after 7 iterations on 34,720 pairs (weights 3540.500 on the right intents, 3540.500 on the others).

The rule (registered thresholds, pack_pins.json newfx rule; group bounds are point bounds) and every clause on the held-out part:

clause	condition	measured	holds
scores S0	the joint fit of T, gamma, eta and kappa succeeded (enough calibration rows with the right intent read and not read, identified, converged, T inside its registered range, every parameter finite); on every held-out row the top value's intent is the fused top-1; the values and the remainder are valid (between zero and one, 0.0 beyond the read intents, adding up to one); the remainder is zero on every list of k intents or fewer	fit yes, top value equals the fused top-1 yes, values valid yes, remainder zero on small lists yes	yes
scores S1	pooled top-label ECE at most 0.03 and its 97.5th one-way percentile at most 0.05	ECE 0.019, 97.5th percentile 0.035	yes
scores S2	top-label ECE at most 0.05 in EN and in LTPL	EN 0.016, LTPL 0.026	yes
scores S3	top-label ECE at most 0.05 in EN_unseen and in LTPL_unseen	EN_unseen 0.032, LTPL_unseen 0.043	yes
scores S4	for each engine of the library, a share of at least 0.99 of the consistency rows has every value and the remainder within 0.01 of the job scorer's, and the ECE from the library's values within 0.01 of the scorer's	auto: 0.999 of the rows within the tolerance, ECE difference -0.0002; eager: 0.999 of the rows within the tolerance, ECE difference -0.0002	yes
scores S5	pooled candidate ECE over the read values at most 0.03 and its 97.5th percentile at most 0.05	ECE 0.005, 97.5th percentile 0.0103	yes
scores S6	pooled remainder ECE at most 0.03 and its 97.5th percentile at most 0.05, and at most 0.05 in each list-size band (6 to 20; 21 to 40; 41 or more intents)	ECE 0.006, 97.5th percentile 0.018; 6 to 20: 0.008; 21 to 40: 0.012; 41 or more: 0.013	yes
check C0	the fit converged and a is positive	converged yes, a 0.274	yes
check C1	pooled ECE at most 0.03 and its 97.5th two-way percentile at most 0.05	ECE 0.014, 97.5th percentile 0.03001	yes
check C2	ECE at most 0.05 in EN and in LTPL	EN 0.026, LTPL 0.03005	yes
check C3	balanced accuracy at 0.5 at least 0.85 and its 2.5th two-way percentile at least 0.8	balanced accuracy 0.873, 2.5th percentile 0.856	yes
check C4	ECE at most 0.05 in EN_unseen and in LTPL_unseen	EN_unseen 0.057, LTPL_unseen 0.015	no
check C5	for each engine of the library, a share of at least 0.99 of the gold and uniform consistency pairs has a probability within 0.01 of the job scorer's, and the ECE from the library's probabilities within 0.01 of the scorer's	auto: 1.000 of the pairs within the tolerance, ECE difference -0.000001; eager: 1.000 of the pairs within the tolerance, ECE difference 0.0000001	yes

Held-out results of scores, pooled and by language group, each with its 95% interval (recall@k: the share of messages whose right intent was read):

rows	count	top-label ECE	candidate ECE	remainder ECE	recall@5	accuracy	mean top value minus accuracy	Brier
all held-out rows	13,405	0.019 [0.012, 0.035]	0.005 [0.004, 0.0103]	0.006 [0.005, 0.018]	0.928 [0.912, 0.942]	0.767 [0.741, 0.791]	-0.014 [-0.031, 0.003]	0.320 [0.290, 0.353]
English (EN)	4,725	0.016 [0.014, 0.035]	0.004 [0.004, 0.0097]	0.006 [0.005, 0.017]	0.951 [0.937, 0.962]	0.820 [0.793, 0.844]	-0.006 [-0.023, 0.012]	0.251 [0.221, 0.285]
Lithuanian and Polish (LTPL)	4,943	0.026 [0.018, 0.046]	0.008 [0.006, 0.014]	0.011 [0.009, 0.025]	0.912 [0.894, 0.928]	0.727 [0.696, 0.756]	-0.023 [-0.043, -0.002]	0.370 [0.336, 0.407]
English, sets outside the seen-domain list (EN_unseen)	2,569	0.032 [0.026, 0.052]	0.011 [0.008, 0.017]	0.012 [0.008, 0.024]	0.953 [0.939, 0.966]	0.807 [0.779, 0.832]	-0.030 [-0.049, -0.010]	0.271 [0.241, 0.305]
Lithuanian and Polish, sets outside the seen-domain list (LTPL_unseen)	3,050	0.043 [0.0296, 0.066]	0.015 [0.0102, 0.022]	0.011 [0.0098, 0.028]	0.921 [0.903, 0.938]	0.736 [0.703, 0.768]	-0.043 [-0.065, -0.019]	0.367 [0.333, 0.406]
Lithuanian (information)	2,530	0.034 [0.026, 0.056]	0.008 [0.007, 0.015]	0.013 [0.0105, 0.0304]	0.918 [0.900, 0.935]	0.728 [0.696, 0.758]	-0.025 [-0.047, -0.002]	0.369 [0.333, 0.406]
Polish (information)	2,413	0.023 [0.021, 0.047]	0.0096 [0.007, 0.016]	0.019 [0.013, 0.036]	0.906 [0.887, 0.924]	0.726 [0.694, 0.756]	-0.020 [-0.042, 0.002]	0.371 [0.336, 0.409]
German and Slovak (information)	3,737	0.025 [0.021, 0.047]	0.007 [0.006, 0.014]	0.0097 [0.008, 0.028]	0.921 [0.899, 0.939]	0.752 [0.719, 0.785]	-0.014 [-0.037, 0.011]	0.342 [0.300, 0.384]

The remainder on lists over 5 intents (held-out rows), pooled and by list size:

intents in the list	rows	share not read	mean remainder	remainder ECE [95% interval]
over 5	13,341	0.072	0.075	0.006 [0.005, 0.018]
6 to 20	2,823	0.045	0.037	0.008 [0.004, 0.034]
21 to 40	4,234	0.051	0.051	0.012 [0.006, 0.024]
41 or more	6,284	0.098	0.108	0.013 [0.011, 0.035]

Held-out results of check (the right intent and uniformly drawn other intents of the same list, half the weight each), pooled and by language group, with the two-way 95% intervals of the pooled values:

pairs	count	ECE	balanced accuracy	right yes rate	right no rate	AUC	Brier
all held-out pairs	66,731	0.014 [0.008, 0.03001]	0.873 [0.856, 0.888]	0.853 [0.832, 0.873]	0.893 [0.863, 0.915]	0.943 [0.930, 0.954]	0.093 [0.083, 0.104]
English (EN)	23,475	0.026 [0.015, 0.043]	0.893 [0.875, 0.908]	0.909 [0.889, 0.927]	0.877 [0.844, 0.903]	0.959 [0.948, 0.968]	0.078 [0.068, 0.090]
Lithuanian and Polish (LTPL)	24,640	0.03005 [0.016, 0.048]	0.859 [0.841, 0.875]	0.825 [0.8001, 0.849]	0.893 [0.860, 0.917]	0.933 [0.917, 0.946]	0.103 [0.092, 0.115]
English, sets outside the seen-domain list (EN_unseen)	12,797	0.057 [0.038, 0.079]	0.867 [0.844, 0.887]	0.916 [0.897, 0.933]	0.819 [0.776, 0.856]	0.948 [0.935, 0.961]	0.095 [0.082, 0.110]
Lithuanian and Polish, sets outside the seen-domain list (LTPL_unseen)	15,175	0.015 [0.014, 0.038]	0.857 [0.835, 0.877]	0.863 [0.841, 0.883]	0.852 [0.811, 0.884]	0.936 [0.918, 0.950]	0.101 [0.089, 0.116]
Lithuanian (information)	12,608	0.026 [0.016, 0.045]	0.859 [0.841, 0.876]	0.830 [0.804, 0.855]	0.888 [0.853, 0.913]	0.932 [0.916, 0.945]	0.103 [0.092, 0.116]
Polish (information)	12,032	0.035 [0.021, 0.054]	0.859 [0.840, 0.875]	0.820 [0.793, 0.846]	0.897 [0.866, 0.922]	0.933 [0.917, 0.947]	0.103 [0.092, 0.115]
German and Slovak (information)	18,616	0.044 [0.025, 0.066]	0.867 [0.845, 0.886]	0.820 [0.787, 0.8503]	0.913 [0.880, 0.939]	0.942 [0.924, 0.956]	0.097 [0.084, 0.112]

The share of yes answers of check that are right when the intent is rare (from the pooled right yes and right no rates): 1 in 2 messages matching, 0.888 [0.862, 0.909]; 1 in 5 messages matching, 0.665 [0.610, 0.715]; 1 in 20 messages matching, 0.295 [0.248, 0.346].

Consistency of the library's engines with the job scorer on the consistency rows (the library's raw scores with the fitted parameters):

engine	scores: rows within the tolerance	ECE, library	ECE, scorer	raw difference (fused units, 99th percentile)	check: pairs within the tolerance	ECE, library	ECE, scorer
auto	3,234 of 3,236	0.027	0.027	0.011	16,180 of 16,180	0.100	0.100
eager	3,233 of 3,236	0.027	0.027	0.0097	16,180 of 16,180	0.100	0.100

Information on DEV (no clause): check on intents close to the right one (the other intents among the 5 read; each row once): false yes rate 0.420 [0.391, 0.449], on the closest other intent alone 0.629 [0.597, 0.662]; check with intents given by their name only: ECE 0.121 [0.105, 0.142], balanced accuracy 0.836 [0.816, 0.853]. strict rows: scores ECE 0.031 [0.021, 0.056], recall@5 0.909 [0.879, 0.934], check ECE 0.028 [0.016, 0.052]; the pools of domain_seen_by_P: scores ECE 0.019 [0.018, 0.048], recall@5 0.920 [0.888, 0.946], check ECE 0.066 [0.043, 0.092]; English clinc_dev: scores ECE 0.0302 [0.026, 0.079], recall@5 0.982 [0.955, 0.998], check ECE 0.105 [0.073, 0.153].

scores on VAL (label sets seen in training, so this overstates transfer) and on the changed messages and name-only label texts of the target frame (held-out intents only): VAL:banking77_train: ECE 0.082 [0.058, 0.116], recall@5 0.955 [0.937, 0.971]; VAL:clinc_bank_validation: ECE 0.0297 [0.026, 0.045], recall@5 0.995 [0.986, 1.000]; VAL:hwu64_train: ECE 0.029 [0.025, 0.058], recall@5 0.977 [0.962, 0.988]; VAL:massive_dev@da-DK: ECE 0.042 [0.025, 0.077], recall@5 0.956 [0.943, 0.968]; VAL:massive_dev@de-DE: ECE 0.044 [0.026, 0.082], recall@5 0.955 [0.940, 0.967]; VAL:massive_dev@el-GR: ECE 0.047 [0.027, 0.090], recall@5 0.951 [0.938, 0.963]; VAL:massive_dev@en-US: ECE 0.025 [0.019, 0.056], recall@5 0.963 [0.938, 0.980]; VAL:massive_dev@es-ES: ECE 0.043 [0.028, 0.079], recall@5 0.952 [0.935, 0.967]; VAL:massive_dev@fi-FI: ECE 0.047 [0.029, 0.086], recall@5 0.952 [0.941, 0.962]; VAL:massive_dev@fr-FR: ECE 0.042 [0.025, 0.080], recall@5 0.954 [0.938, 0.968]; VAL:massive_dev@hu-HU: ECE 0.059 [0.033, 0.096], recall@5 0.950 [0.938, 0.962]; VAL:massive_dev@it-IT: ECE 0.040 [0.025, 0.075], recall@5 0.963 [0.950, 0.974]; VAL:massive_dev@lv-LV: ECE 0.040 [0.029, 0.079], recall@5 0.939 [0.922, 0.956]; VAL:massive_dev@nb-NO: ECE 0.029 [0.018, 0.066], recall@5 0.963 [0.951, 0.974]; VAL:massive_dev@nl-NL: ECE 0.041 [0.027, 0.075], recall@5 0.953 [0.941, 0.966]; VAL:massive_dev@pl-PL: ECE 0.040 [0.024, 0.080], recall@5 0.961 [0.944, 0.976]; VAL:massive_dev@pt-PT: ECE 0.045 [0.025, 0.083], recall@5 0.960 [0.948, 0.970]; VAL:massive_dev@ro-RO: ECE 0.045 [0.029, 0.084], recall@5 0.953 [0.938, 0.966]; VAL:massive_dev@ru-RU: ECE 0.035 [0.024, 0.065], recall@5 0.963 [0.952, 0.974]; VAL:massive_dev@sl-SL: ECE 0.055 [0.033, 0.098], recall@5 0.946 [0.931, 0.960]; VAL:massive_dev@sv-SE: ECE 0.035 [0.025, 0.067], recall@5 0.962 [0.949, 0.973]; email: ECE 0.054 [0.037, 0.077], recall@5 0.945 [0.927, 0.960]; names: ECE 0.041 [0.027, 0.070], recall@5 0.906 [0.879, 0.930]; nodia: ECE 0.047 [0.0297, 0.068], recall@5 0.919 [0.901, 0.936]; typo1: ECE 0.031 [0.018, 0.051], recall@5 0.929 [0.913, 0.944].

Appendix B lists every number of the record, per pool and per pool family included.

Training (PROTOCOL E1, E2, E5)

B's recipe (the v1 (a) cross-encoder recipe from BAAI/bge-reranker-v2-m3, 2 epochs; then the ONE-0 adapter recipe of v1.1, trained on this model's own data) on data pub (E1); the honest part is data 1.2 and ML 1.2, because probe B12 did not fire (its LTPL against Z on the matched rows was above the -0.010 margin; PROTOCOL entry 2026-10-09 02:28). Z was trained on data v2; that difference is inside what NW-P checks. One training seed (seed 0), against the 3 seeds of PROTOCOL Metrics; further P seeds are DEV information only and never replace this model. The checkpoint was selected on clinc_dev (the cross-encoder alone over the frozen bge-m3 top 8, rows not flagged near_test), a pool in domain_seen_by_P, among the evaluations from the end of epoch 0 on (selected: step 6400). Message changes in the cross-encoder groups (the shares of this model's record): diacritic stripping 0.3 of the lt and pl groups; one e-mail wrapper (de 0.3, en 0.3, lt 0.2, pl 0.2); one typo (en 0.1, lt 0.1, pl 0.1). Clean floor of E5 (5): per language at least half of the cross-encoder groups stay clean; nominal clean shares of this model: de 0.7, en 0.63, lt 0.504, pl 0.504. For lt and pl the e-mail wrapper share is 0.2 instead of 0.3, the price of keeping half of the lt and pl groups clean, although the probe's e-mail loss was larger in lt and pl than in English. Retrieval adapter (V-ADIA): stripping 0.3 of the lt and pl messages; one e-mail wrapper (de 0.3, en 0.3, lt 0.2, pl 0.2); one typo (en 0.1, lt 0.1, pl 0.1). Label-text mix of the honest groups (V-MIX): v2 0.4, v3s-A 0.125, v3s-B 0.125, without examples 0.2, name and first sentence 0.05, name only 0.1; closed-inventory groups 0.6 inference text and 0.4 that mix. E-mail wrappers (PROTOCOL E5 (4d)): the chosen lt and pl wrappers were read before the P job, and it was decided to strike those whose words state a request, a topic or a customer fact (PROTOCOL 2026-10-11 21:55; strike file sha256 9444ab6adc07864e62848d27b0c8826660a2478043c28fb967134ebfeed3ce26): lt 11 of 200 and pl 23 of 200 struck, each replaced by the next kept wrapper; wrappers used per language: en 200, lt 200, pl 200, de 182. On the v1.1 probe's LTPL rows the retrieval recall fell from 0.943 to 0.909 with diacritics removed; this model's retrieval adapter is trained also on stripped, wrapped and typo messages (V-ADIA). No KD teacher was used: P_full has no KD term, because the teacher probe of PROTOCOL E5 (6) chose no teacher (PROTOCOL 2026-10-11 12:46, result of the teacher choice); the embedding target of the retrieval adapter, BAAI/bge-m3 (the embedding teacher of NOTICE item 2), is not a KD teacher.

Contamination statement (PROTOCOL E3 as amended by E5 (13))

This model was trained on public data, including the official training splits of Banking77, HWU64, MASSIVE 1.1 (which contains the text collection HWU64 was built from) and CLINC150 banking and credit_cards; no test split was used; every training message within exact match or bge-m3 cosine 0.90 of any test message was removed (removed English training rows by the audited set of the nearest test message: Banking77 test 7,827; CLINC150 banking and credit_cards test 1,671; HWU64 test 2,273; MASSIVE 1.1 en-US test 3,904; xSID en 255; and 1,480 rows by the registered English audit of MInDS-14, MTOP en, SIB-200 eng_Latn and xSID en; cross-lingually 13,439 English rows excluded (P0: 13,027) and 5,441 MASSIVE utterance ids dropped in all 51 locales; lever rows of L1 removed: v37 8 (from 12 hits over the audited files; a row can hit several); V-IND rows 70 English and 161 cross-lingual (Appendix A), E4); MTOP, xSID, MInDS-14 and SIB-200 are benchmarks whose data the model never saw (their domains are partly covered: banking for MInDS-14; alarms, reminders and weather for xSID; assistant, calling and messaging domains for MTOP); all 51 MASSIVE locales are in its training data; in-domain label texts whose quoted examples match a test message were trained without those examples; base models BAAI/bge-reranker-v2-m3 and bge-m3, whose training-data statements do not mention Banking77, HWU64, MASSIVE, SLURP, SNIPS, xSID or MTOP (MInDS-14, SIB-200 and CLINC150 were not part of that check; web-crawl exposure cannot be excluded); it also trains on synthetic customer-service data: a banking, fintech and insurance pack generated by Qwen2.5-14B-Instruct from service names written for this project, the further synthetic domains of V-37 (32 when 5 are held out) generated by Qwen2.5-7B-Instruct, and packs of 8 non-bank industries (4 more held out), including billing and payment intents, generated by Qwen2.5-14B-Instruct from industry names written for this project; the pretraining data of these generators is not disclosed; near copies of test messages were removed as above.

Which benchmark families this model saw (PROTOCOL E3 display rule)

test family	T3 saw this family's training split?
Banking77	yes
HWU64	yes (also through MASSIVE's text collection)
MASSIVE 1.1	yes
MTOP de/fr/es	no (domain partly seen)
xSID lt	no
MInDS-14	no
SIB-200	no
CLINC150 banking and credit_cards	yes

T1 (zero-shot), FS (few-shot) and T3 (this model) numbers appear in one table only in separate, labeled columns, never inside a zero-shot claim, mean or comparison. P's Banking77 number stands next to third-party Banking77 numbers only in a protocol-matched row 'training data available'.

Disclosures

Result-informed choices: the composition of P_full, the exclusions of E5 (11), the B12 rule, the clean floor and the checkpoint rule were fixed after the v1.1, stage-1.2, seeds, noise-probe and model D DEV results and after the stage-1 test aggregates of 2026-10-05 were known; none uses a test row; that check ships as a beta was decided after its DEV calibration result (Calibrated outputs check and scores).
Model D, whose data overlaps P_full's (data 1.2, the Bitext banking verticals, NLU++ banking, MultiDoGO and the D pack), was below Z on matched DEV rows by 0.0131 on LTPL and 0.0152 on clinc_dev, and by 0.0086 on EN_DEC without domain_seen_by_D (PROTOCOL 2026-10-07 17:56).
MASSIVE 1.1 has no Lithuanian locale: P's Lithuanian training text is machine-translated or generated by EuroLLM-9B-Instruct-2512, and every Lithuanian and Polish DEV number is on machine-translated text.
Text written for this project: the frozen data v2 domain list, the 9 D-pack seeds and the 12 V-IND industry names are prompt inputs and label metadata, not training text, but they shape label texts (467 of the 1,709 synthetic training label texts of data 1.2, 0.273, contain their domain name); the PRESTO domain map was chosen the same way (metadata).
MASSIVE's 50 non-English locales and PRESTO were chosen knowing that MASSIVE and MTOP were low in the stage-1 test aggregates, by license and domain coverage, not from any test row; PRESTO covers MTOP's calling and messaging domains (MTOP is 'domain partly seen'); PRESTO was added by the decision of 2026-10-06 15:30; P trains on all 51 MASSIVE locales, so the held-out-locale group of the zero-shot track has no meaning for P; model D and P share Bitext, NLU++ banking and MultiDoGO; DSTC11 banking and finance and ABCD are model D's only; the D pack is shared by model D and P_full (never P0).
PROTOCOL E5 (4c) made no V-IND pack for travel, hotels, health care or HR and IT, but P's other training data cover parts of travel, hotels and health care: among others, its training domains include 'travel', 'hospitality', 'hospital', 'telehealth service' and 'medical laboratory' (data 1.2, the honest part); 'travel' and 'health and fitness' (PRESTO) (data pub manifest pub_training_domains). nlupp_hotels_dev, mixed_dev and clinc_dev (whose work domain covers HR) are in domain_seen_by_P (PROTOCOL E5 (7)), so they are not part of NW-P's unseen groups. The 5 held-out V-37 domains and the 4 held-out V-IND industries are DEV pools only.
Where PROTOCOL cites statements made in Russian, it gives them in English translation.
What survives E4 (audit of P's training composition, PROTOCOL 2026-10-11 12:18), on one basis: the training rows of this model that remain after all of E4, including the trainer's exclusion of English rows hit cross-lingually (data pub's trained rows of a source less its rows on ML pub's exclusion list, by source as registered in PROTOCOL 2026-10-11 22:03). Of the 8,990 Banking77 train rows left after the 10% VAL carve-out, 2,050 remain (about 23%: 2,070 trained less 20 on the exclusion list); of the 8,045 HWU64 train rows left after it, 2,640 remain (about 33%: 3,904 trained less 1,264 on the exclusion list); of the 2,999 CLINC banking and credit_cards train rows, 1,591 remain (about 53%: 1,612 trained less 21 on the exclusion list); of the 11,514 MASSIVE en-US utterance ids, 1,343 remain (about 12%: data pub's 6,771 trained ids less the 5,428 whose en-US rows the trainer excludes, as the chain removes a hit id in every locale). The further English exclusions of P_full (412 rows: those of L1's and L2's lists and the new rows of the cross-lingual re-audit) are all lever rows (row ids from 171,125 on, where the lever rows start), so these counts are P0's as well. These shares replace those of the entries of 2026-10-11 12:18 and 22:03, which counted the HWU64 and CLINC banking and credit_cards shares (and, at 22:03, the Banking77 count) before the trainer's exclusion. ML pub keeps 142 MASSIVE lt translations, against 1,368 at its 24 rows per intent for the 57 trained MASSIVE labels.
Eight labels of the frozen closed inventories have no training row (Banking77 exchange_rate and getting_virtual_card, two HWU64 labels, three MASSIVE labels, CLINC pay_bill: six left by dev wins, two by E4 and the minimum-rows rule) and are not used as negatives (PROTOCOL 2026-10-11 12:18 (c)). VAL leaves out their 3,061 VAL rows and scores each VAL set against its trained labels only (Banking77 75, HWU64 62, MASSIVE 57 and CLINC banking and credit_cards 29 labels), so every VAL figure in this card is VAL without these 8 labels; no VAL figure with them was computed, because those rows were never scored. PROTOCOL 2026-10-11 12:18 (c) planned VAL figures also with these labels; that plan is amended, and disclosed, in the PROTOCOL entry of the PACK build of this release.
In training, 42 of the 75 trained Banking77 labels (and 15 HWU64, 15 MASSIVE and 10 CLINC labels) use their label text without the quoted examples by the quoted-example rule of E2 (3), while inference uses the frozen text (PROTOCOL 2026-10-11 12:18).
The 12 V-IND industries (a judgment, not a measurement), the V-E prompts, the industry exclusions and the probe objects were written knowing the registered label inventories, the stage-1 test aggregates and the model D DEV result; the 9 D-pack seeds were written before model D existed. They are chosen to avoid the registered label sets; Qwen2.5 pretraining may include public intent data; the 0.90 filter removes near copies only; the money-side intents of V-IND and the D pack sit close in meaning to banking labels.
The e-mail target uses one fixed probe wrapper per language written for this project; the wrapper filter (e)(i) and the prompt were written knowing its words; real e-mails with quoted threads, disclaimers or long bodies are not covered; EuroLLM-9B-Instruct-2512 is not aligned to human preferences, and the wrappers were generated in fp16.
The fresh DEV pools of V-37 and V-IND are synthetic, made by the generators of the synthetic training data.

Test results

P's single test evaluation (PROTOCOL E3 (a), (b); E5 (10e)) ran once, on the frozen files of this release, with the registered runtime and no library pre-processing (PROTOCOL 2026-10-13 23:31; result and E3 aggregate 2026-10-14 13:27). It is a second, disclosed test exposure of the registered sets after the single look and selected nothing. The package it read has the sha256 ba3f461ab703394cfc362fa330c2aa890cf2818f5412991a61d74186733b353b by the evaluation's rule; this card was rebuilt after it, and only README.md and r2intent_pack_manifest.json changed: the model files are the evaluated ones.

Every number in this section is track T3: trained on public data including the training splits of Banking77, HWU64, MASSIVE and CLINC150 banking and credit_cards. P is not a zero-shot model.

Set	T3 accuracy [95% interval]	T3 saw this family's training split?
Banking77	0.810 [0.797, 0.824]	yes
HWU64	0.860 [0.838, 0.879]	yes (also through MASSIVE's text collection)
MASSIVE, 17 languages	0.774 [0.761, 0.785]	yes
MTOP, German, French and Spanish	0.633 [0.624, 0.642]	no (domain partly seen)
xSID, Lithuanian	0.888 [0.860, 0.914]	no
MInDS-14, 9 languages	0.896 [0.888, 0.905]	no
SIB-200, 9 languages	0.715 [0.690, 0.738]	no

T3 7-set mean (the E3 headline: the unweighted mean of the 7 set aggregates, registered bootstrap, 2,000 resamples, seed 0): 0.796 [0.789, 0.804]. It covers 21 languages; the model's 51 training locales are a training fact, not an evaluation claim.

T3 held-out mean (MTOP de, fr and es; xSID lt; MInDS-14 9 non-English; SIB-200 9 languages): 0.783 [0.773, 0.793]; their domains are partly covered: banking for MInDS-14; alarms, reminders and weather for xSID; assistant domains for MTOP through MASSIVE, and calling and messaging through PRESTO, so MTOP is domain partly seen.
In-domain mean (Banking77, HWU64, MASSIVE): 0.814 [0.805, 0.824].
English sets mean (Banking77, HWU64): 0.835 [0.822, 0.847].
Multilingual sets mean (MASSIVE, MTOP, xSID, MInDS-14, SIB-200): 0.781 [0.772, 0.789].

CLINC150 banking and credit_cards, reported separately and never in a mean, with Banking77 0.810 beside it: 0.949 (n 900; row interval [0.934, 0.963], label-cluster interval [0.923, 0.970]); banking 0.931 (n 450); credit_cards 0.967 (n 450).

With the quoted examples stripped from the frozen descriptions (p_noex, information, never in an aggregate): 7-set mean 0.784 [0.778, 0.791]; paired p_noex minus p: headline -0.012 [-0.016, -0.008]; held-out mean -0.014 [-0.019, -0.009]; in-domain mean -0.009 [-0.015, -0.003]; English sets mean -0.007 [-0.016, +0.002]; multilingual sets mean -0.014 [-0.018, -0.010].

Per set and locale (T3; accuracy with its row interval, macro-F1, recall@5, and accuracy with the quoted examples stripped):

Set	Locale	Messages	Accuracy [95% interval]	Macro-F1	Recall@5	Accuracy, examples stripped
banking77	en	3,076	0.810 [0.797, 0.824]	0.807	0.958	0.798
clinc_bank	en	900	0.949 [0.934, 0.963]	0.949	0.991	0.946
hwu64	en	1,076	0.860 [0.838, 0.879]	0.848	0.969	0.857
xsid	lt	500	0.888 [0.858, 0.916]	0.789	0.990	0.886
xsid	en	500	0.958 [0.940, 0.974]	0.934	0.996	0.938
xsid	de	500	0.944 [0.924, 0.964]	0.856	0.988	0.926
xsid	it	500	0.940 [0.920, 0.960]	0.859	0.996	0.922
xsid	nl	500	0.930 [0.908, 0.952]	0.912	0.980	0.910
xsid	da	500	0.940 [0.918, 0.960]	0.864	0.988	0.924
massive	en-US	2,974	0.810 [0.796, 0.824]	0.738	0.943	0.798
massive	pl-PL	2,974	0.782 [0.767, 0.797]	0.702	0.946	0.766
massive	lv-LV	2,974	0.753 [0.737, 0.768]	0.693	0.933	0.733
massive	de-DE	2,974	0.784 [0.770, 0.799]	0.717	0.945	0.771
massive	fr-FR	2,974	0.782 [0.767, 0.796]	0.714	0.942	0.770
massive	es-ES	2,974	0.778 [0.763, 0.792]	0.709	0.943	0.763
massive	it-IT	2,974	0.780 [0.765, 0.796]	0.708	0.946	0.769
massive	ru-RU	2,974	0.770 [0.755, 0.785]	0.699	0.944	0.761
massive	nl-NL	2,974	0.781 [0.765, 0.795]	0.700	0.945	0.769
massive	pt-PT	2,974	0.775 [0.759, 0.789]	0.711	0.944	0.755
massive	ro-RO	2,974	0.771 [0.756, 0.785]	0.702	0.939	0.765
massive	sv-SE	2,974	0.797 [0.782, 0.811]	0.725	0.953	0.783
massive	da-DK	2,974	0.779 [0.765, 0.794]	0.698	0.946	0.763
massive	nb-NO	2,974	0.782 [0.768, 0.797]	0.710	0.947	0.775
massive	fi-FI	2,974	0.768 [0.753, 0.783]	0.693	0.941	0.753
massive	hu-HU	2,974	0.751 [0.735, 0.767]	0.679	0.939	0.740
massive	sl-SL	2,974	0.757 [0.741, 0.772]	0.684	0.943	0.743
massive	el-GR	2,974	0.761 [0.746, 0.776]	0.684	0.931	0.746
sib200	lit_Latn	1,004	0.711 [0.683, 0.738]	0.709	0.977	0.710
sib200	pol_Latn	1,004	0.717 [0.689, 0.744]	0.717	0.983	0.729
sib200	lvs_Latn	1,004	0.708 [0.680, 0.734]	0.708	0.988	0.723
sib200	est_Latn	1,004	0.693 [0.664, 0.720]	0.700	0.984	0.693
sib200	deu_Latn	1,004	0.730 [0.703, 0.757]	0.729	0.976	0.737
sib200	fra_Latn	1,004	0.725 [0.695, 0.751]	0.725	0.979	0.735
sib200	spa_Latn	1,004	0.719 [0.691, 0.746]	0.723	0.977	0.719
sib200	fin_Latn	1,004	0.710 [0.682, 0.737]	0.709	0.977	0.722
sib200	eng_Latn	1,004	0.719 [0.692, 0.747]	0.718	0.984	0.735
minds14	pl-PL	562	0.849 [0.819, 0.877]	0.828	0.975	0.865
minds14	de-DE	611	0.863 [0.835, 0.889]	0.827	0.972	0.884
minds14	fr-FR	539	0.935 [0.915, 0.955]	0.934	0.978	0.935
minds14	es-ES	486	0.881 [0.852, 0.909]	0.854	0.988	0.874
minds14	it-IT	696	0.925 [0.905, 0.944]	0.923	0.968	0.934
minds14	ru-RU	539	0.917 [0.892, 0.939]	0.914	0.981	0.928
minds14	cs-CZ	574	0.897 [0.871, 0.922]	0.863	0.960	0.899
minds14	nl-NL	654	0.873 [0.847, 0.896]	0.871	0.960	0.887
minds14	pt-PT	604	0.924 [0.902, 0.945]	0.916	0.980	0.932
minds14	en-US	563	0.918 [0.895, 0.940]	0.919	0.988	0.918
mtop	en	4,386	0.638 [0.624, 0.651]	0.497	0.889	0.580
mtop	de	3,549	0.618 [0.602, 0.634]	0.433	0.882	0.539
mtop	fr	3,193	0.631 [0.615, 0.648]	0.455	0.867	0.565
mtop	es	2,998	0.649 [0.631, 0.666]	0.429	0.874	0.582

Test information of check and scores (PROTOCOL 2026-10-13 00:06 (11)): the frozen calibration file of this release applied to the same model's scores after the registered outputs were written; nothing was fitted or changed after these numbers. Per set, pooled over its locales, 95% intervals by 2,000 resamples, seed 0, 15 bins; no mean over sets. For an ECE the interval can lie above the value, because an ECE recomputed on resamples runs higher.

scores:

Set	Messages	Top-label ECE	Candidate ECE	Mean top value minus accuracy	Recall@5	Remainder ECE (lists over 5 intents)	Brier
Banking77	3,076	0.088 [0.061, 0.113]	0.020 [0.016, 0.027]	-0.087 [-0.112, -0.061]	0.958 [0.944, 0.969]	0.074 [0.061, 0.088]	0.293 [0.251, 0.340]
CLINC150 banking and credit_cards	900	0.025 [0.020, 0.044]	0.009 [0.007, 0.015]	-0.022 [-0.036, -0.006]	0.991 [0.980, 0.999]	0.012 [0.010, 0.019]	0.079 [0.049, 0.114]
HWU64	1,076	0.033 [0.027, 0.060]	0.007 [0.008, 0.016]	-0.025 [-0.049, -0.001]	0.969 [0.953, 0.983]	0.024 [0.018, 0.040]	0.206 [0.160, 0.259]
xSID (6 locales)	3,000	0.023 [0.015, 0.040]	0.009 [0.006, 0.015]	-0.021 [-0.037, -0.006]	0.990 [0.978, 0.999]	0.005 [0.002, 0.013]	0.102 [0.051, 0.178]
MASSIVE (18 locales)	53,532	0.030 [0.013, 0.052]	0.013 [0.006, 0.022]	+0.029 [+0.009, +0.052]	0.943 [0.924, 0.956]	0.002 [0.003, 0.014]	0.313 [0.253, 0.380]
SIB-200 (9 languages)	9,036	0.023 [0.020, 0.065]	0.009 [0.008, 0.024]	+0.001 [-0.051, +0.055]	0.981 [0.963, 0.996]	0.001 [0.000, 0.016]	0.408 [0.277, 0.530]
MInDS-14 (10 locales)	5,828	0.030 [0.021, 0.055]	0.011 [0.007, 0.022]	-0.022 [-0.052, +0.006]	0.974 [0.957, 0.989]	0.008 [0.003, 0.021]	0.143 [0.070, 0.228]
MTOP (4 locales)	14,126	0.065 [0.039, 0.131]	0.018 [0.011, 0.036]	-0.033 [-0.114, +0.070]	0.879 [0.792, 0.937]	0.078 [0.047, 0.131]	0.517 [0.380, 0.661]

check (a beta: it shipped in v1 after it failed one pre-registered DEV clause, C4), each message with its right intent and one other intent of the set drawn at random:

Set	Messages	ECE	Balanced accuracy at 0.5	True-yes rate	True-no rate	False-yes rate on the non-gold top 5	AUC	Brier
Banking77	3,076	0.056 [0.046, 0.072]	0.941 [0.926, 0.956]	0.980 [0.974, 0.985]	0.903 [0.872, 0.931]	0.768 [0.710, 0.823]	0.988 [0.984, 0.992]	0.047 [0.037, 0.057]
CLINC150 banking and credit_cards	900	0.036 [0.035, 0.054]	0.958 [0.937, 0.974]	0.963 [0.944, 0.981]	0.952 [0.910, 0.982]	0.296 [0.236, 0.357]	0.991 [0.985, 0.996]	0.034 [0.023, 0.046]
HWU64	1,076	0.083 [0.062, 0.106]	0.917 [0.891, 0.939]	0.849 [0.799, 0.892]	0.984 [0.963, 0.997]	0.214 [0.169, 0.265]	0.987 [0.977, 0.994]	0.061 [0.048, 0.078]
xSID (6 locales)	3,000	0.115 [0.045, 0.190]	0.865 [0.790, 0.913]	0.763 [0.624, 0.877]	0.967 [0.908, 0.996]	0.133 [0.051, 0.262]	0.970 [0.937, 0.990]	0.098 [0.063, 0.153]
MASSIVE (18 locales)	53,532	0.114 [0.087, 0.147]	0.880 [0.842, 0.913]	0.776 [0.700, 0.840]	0.985 [0.974, 0.993]	0.183 [0.139, 0.232]	0.983 [0.975, 0.988]	0.084 [0.064, 0.108]
SIB-200 (9 languages)	9,036	0.364 [0.340, 0.479]	0.569 [0.448, 0.585]	0.148 [0.063, 0.253]	0.991 [0.973, 0.999]	0.012 [0.005, 0.018]	0.841 [0.811, 0.879]	0.331 [0.304, 0.431]
MInDS-14 (10 locales)	5,828	0.049 [0.030, 0.073]	0.918 [0.874, 0.956]	0.921 [0.864, 0.963]	0.915 [0.852, 0.969]	0.246 [0.147, 0.360]	0.969 [0.943, 0.987]	0.064 [0.040, 0.093]
MTOP (4 locales)	14,126	0.132 [0.089, 0.183]	0.846 [0.790, 0.887]	0.721 [0.609, 0.808]	0.970 [0.949, 0.985]	0.415 [0.289, 0.548]	0.955 [0.926, 0.976]	0.114 [0.083, 0.159]

On every message of every set the recomputed top intent equals the registered prediction and the recomputed top 5 equal the stored ones.

The other models of the front table were run without training examples of the intents: r2intent-zs-v1 (Z) and Qwen3-Embedding-8B from the single look (PROTOCOL 2026-10-08 01:35), EmbeddingGemma 2 and Laya from the post-look baselines registered before the look (03:05); New topics from the DEV comparison of 2026-10-08 14:19, and for this model from its stored DEV scores on the same rows (the P job's record). The front table puts them next to this model's T3 numbers in one table with the best number of each row in bold; the registered display rule (E3) keeps T3 numbers in separate labelled columns, as in this section, and the front table's form is logged in PROTOCOL.

License

CC BY-NC 4.0 (LICENSE). Commercial use needs a separate written license from the copyright holder. For commercial use, contact lukashcode@gmail.com or info@liriti.com. A company can also ask there for a free 30-day evaluation, to test R2Intent v1 on its own messages before buying. Third-party notices: NOTICE. Copyright © 2026 Lukaš Patrik Magalinski.

Appendix A: removal counts (E4)

Section 8 of the data notebook, by the audited set of the nearest test message: Banking77 test 7,827, CLINC150 banking and credit_cards test 1,671, HWU64 test 2,273, MASSIVE 1.1 en-US test 3,904, xSID en 255
Named pairs of E4, each separately (English training rows removed by section 8; data pub manifest pub_e4_named_pairs.section8_removed_by_source_and_set, which counts every removed row by source and set, so a pair it does not list removed no row): MASSIVE train vs the HWU64 test 1,024; MASSIVE train vs the MASSIVE en-US test 1,659; HWU64 train vs the MASSIVE en-US test 1,830; HWU64 train vs the HWU64 test 1,070; Banking77 train vs the Banking77 test 6,669; NLU++ banking vs the Banking77 test 36; NLU++ banking vs the CLINC banking test 7; MultiDoGO vs the Banking77 test 194; MultiDoGO vs the CLINC banking test 144; CLINC banking and credit_cards train vs the confirmatory test 1,151; PRESTO vs the MASSIVE en-US test 34; PRESTO vs xSID en (section 8) 0. Pairs with a file of the registered English audit, hits per file (a row can hit several files; their removed rows are in the audit's rows by source below): NLU++ banking vs MInDS-14: minds14 en-US 5, minds14 en-GB 8, minds14 en-AU 5; MultiDoGO vs MInDS-14: minds14 en-US 66, minds14 en-GB 119, minds14 en-AU 69; PRESTO vs MTOP en: mtop en train 55, mtop en eval 18, mtop en test 20; PRESTO vs xSID en: xsid en train 14, xsid en valid 0.
Registered English audit (MInDS-14, MTOP en, SIB-200 eng_Latn and xSID en, every audited file; exact match or bge-m3 cosine 0.90; data pub manifest pub_registered_test_audit): 1,480 training rows removed, by source: massive_train 468, clinc150 320, bitext_dom 179, hwu64_train 164, multidogo 146, presto 80, bitext 46, clinc_bank_train 24, synthetic_qwen 23, banking77_train 18, nlupp_banking 10, hint3 1, multiwoz22 1. Hits per audited file (a row can hit several files, so hits are not removed rows): minds14 en-US 194, minds14 en-GB 241, minds14 en-AU 224, mtop en train 695, mtop en eval 228, mtop en test 294, sib200 eng_Latn train 0, sib200 eng_Latn dev 0, sib200 eng_Latn test 0, xsid en train 633, xsid en valid 26; summed per family: minds14 659, mtop en 1,217, sib200 eng_Latn 0, xsid en 659.
MInDS-14 label-name filter of the registered English audit (data pub manifest pub_registered_test_audit.minds14_name_filter and dropped.rows_by_stage_reason): synthetic label names at bge-m3 cosine 0.85 or more to a MInDS-14 label name; 1 of 1,709 synthetic labels dropped, with 17 training rows.
English rows the trainer excludes after the cross-lingual audits of E4: 13,439 for this model, that is 13,027 by ML pub's audit (P0's list), 6 more by L1's audit (13,033) and 1 more by L2's audit (13,034, the list of the top layer L2), and the 405 new rows of the cross-lingual re-audit of the lever rows (list sha256 63778b8a187bccea31ffb3742d3243239b108442d96493d7afc8fd8e84404924: 408 English rows, 3 of them already listed; with them the re-audit removes the 424 multilingual rows of their translation groups (lt 212, pl 212)) (PROTOCOL 2026-10-11 22:03).
Per source, the 13,027 English rows of ML pub's exclusion list (P0's list and the first part of this model's list; PROTOCOL 2026-10-11 22:03): 4,526 from data 1.2's own sources (bitext 2,446, clinc150 1,201, synthetic_qwen 712, hint3 89, multiwoz22 78) and 8,501 from the public sources (massive_train 5,428, hwu64_train 1,264, bitext_dom 869, presto 835, multidogo 45, clinc_bank_train 21, banking77_train 20, nlupp_banking 19).
Wrapper pair of E4 (no e-mail wrapper string at exact match or bge-m3 cosine 0.90 to a test message; filter (i) of E5 (4d)), for the wrapper file of this model's record (sha256 c48669d38f5fd39b9014e97439dde15649c547fc35bcabaa3e807683d3d55170; counts file ve_wrap_counts.json sha256 66ed0b86a62629fed69841e25954ccddc57e3fc426191c7a76356cdba1053d0f, PROTOCOL 2026-10-11 15:22): wrappers removed as test matches en 41 (exact 0, cosine 41), lt 0, pl 29 (exact 23, cosine 6), de 6 (exact 0, cosine 6) (the matched test messages are MASSIVE test messages); removed by the DEV rule: en 2 (hint3_dev).
Cross-lingual and chained (ML pub's multilingual manifest, its values as recorded; its English list is P0's): {"english_rows_hit_cross_lingually": 9944, "english_rows_excluded_by_the_trainer": 13027, "massive_ids_dropped_in_every_locale": 5441, "rows_dropped_by_kind": {"human": 268868, "train": 5314, "native": 58}}
Lever rows of L1 (data_publevers manifest publevers.leakage and publevers.rules; exact match or bge-m3 cosine 0.90): section 8 sets: no hit; registered English audit, hits per audited file: minds14 en-US 1 (exact 0; v37 1); minds14 en-AU 1 (exact 0; v37 1); mtop en train 5 (exact 0; v37 5); xsid en train 5 (exact 0; v37 5); no hit in the 7 other audited files. Rows removed (a row can hit several files, so hits are not removed rows): section 8 sets dpack 0, v37 0; registered English audit dpack 0, v37 8; MInDS-14 label-name filter dpack 0, v37 0.
V-IND rows of L2 (data_indpack manifest leakage.indpack, as recorded): {"english_removed": 70, "cross_lingual_removed": 161, "banking77_train_audit_removed": 51, "english_by_source_family_split_rule": {"synthetic_qwen|xsid|train|cos_ge_0.90": 4, "synthetic_qwen|minds14|train|cos_ge_0.90": 5, "synthetic_qwen|banking77|test|cos_ge_0.90": 13, "synthetic_qwen|banking77|train|cos_ge_0.90": 42, "synthetic_qwen|clinc_confirm|test|cos_ge_0.90": 3, "synthetic_qwen|massive|train|cos_ge_0.90": 1, "synthetic_qwen|hwu64|train|cos_ge_0.90": 2}}
Removed English training rows per source and rule (exact or cosine >= 0.95; 0.90 to 0.95): banking77_train 3,623 and 3,069; bitext 17 and 227; bitext_dom 80 and 817; clinc150 228 and 388; clinc_bank_train 523 and 712; hint3 2 and 16; hwu64_train 1,730 and 1,232; massive_train 1,319 and 1,485; multidogo 57 and 281; multiwoz22 0 and 3; nlupp_banking 7 and 36; presto 6 and 72
Named pairs of E4 in the multilingual manifest (drop pairs: pairs of a training row and a test message over all languages and rules, not rows; ML pub manifest pub_leakage.named_pairs): banking77_train vs banking77 test 361; clinc_bank_train vs the confirmatory test 77; hwu64_train vs hwu64 test 23; hwu64_train vs massive test 19,505; massive_train vs hwu64 and massive tests 2,808,220; nlupp_banking, multi3nlupp_es, multidogo vs banking77, minds14, clinc banking tests 232; presto vs mtop, massive, xsid tests 10,558
Registered label sets against P's training label names (name cosine 0.80 or a word-set relation; by design for Banking77, HWU64, MASSIVE and CLINC banking): banking77 77 of 77 seen; clinc_confirm 30 of 30 seen; hwu64 63 of 64 seen; massive_en 58 of 59 seen; sib200_lit 2 of 7 seen; xsid_en 11 of 16 seen; xsid_lt 10 of 15 seen
Rule: section 8: exact or bge-m3 cosine >= 0.90 to an audited test message (Banking77, HWU64, MASSIVE en-US, xSID en and lt, SIB-200 lit_Latn, the confirmatory CLINC set); counted per training source and the audited set of the nearest message
Base-model data statements (PROTOCOL 2026-10-04, after step 0b): the training-data statements of BAAI/bge-m3 and BAAI/bge-reranker-v2-m3 do not mention Banking77, HWU64, MASSIVE, SLURP, SNIPS, xSID or MTOP; exposure through web-crawl pre-training text cannot be excluded.
The label sets of Banking77, HWU64, MASSIVE and CLINC150 banking and credit_cards are P's training label sets by design (label-name overlap with the registered sets); the full per-source, per-set and per-rule counts, the named pairs and the label-name overlap are in the data pub and ML pub manifests listed below.

Appendix B: every number of the calibration record of check and scores

Every number of newfx_calibration_20261013T080704Z.json (sha256 a7ebb3ff746cc9ac89925d192a1ac9cfd2b943235f8641b6df2c331b5a55e7b0), and every flag, as the record holds it, part by part; a path names the keys from the top of the record, n/a is a value the record leaves empty (not finite or not given), and a pair in brackets is a 95% interval. Strings (file names, notes) are left out. An interval of a pool, a pool family, a VAL key or a target is taken over the resamples (of 2,000) whose value is finite, and interval_resamples_not_finite gives how many were left out (a blank cell there: none was left out, or the metric has no value); with few clusters an interval can have zero width, and then it says little. Values such as 0.9999999999999998 and 1.0000000000000002 are 1 up to floating-point rounding; they are printed as the record holds them.

set-up, inputs, rule and outcome

path	value
standin	no
copied_block.equal	yes
library_content_sha256.equal	yes
library_content_sha256.all_files_equal_information	yes
inputs.job_record.gate_pass	yes
inputs.dev_pinned	yes
d22.note	n/a
d22.evidence	n/a
realised_counts.registered.p_dev_pools	94
realised_counts.registered.p_dev_rows	24,044
realised_counts.registered.p_dev_decision_rows	23,744
realised_counts.registered.kept_rows	20,486
realised_counts.registered.kept_pools	85
realised_counts.registered.base_pools	26
realised_counts.registered.kept_clusters	644
realised_counts.registered.kept_messages	8,530
realised_counts.registered.calibration_rows	7,081
realised_counts.registered.calibration_clusters	223
realised_counts.registered.heldout_rows	13,405
realised_counts.registered.heldout_clusters	421
realised_counts.registered.heldout_messages	5,507
realised_counts.registered.strict_rows_calibration	2,748
realised_counts.registered.strict_rows_heldout	5,880
realised_counts.registered.check_pairs_calibration	34,720
realised_counts.registered.check_gold_calibration	7,081
realised_counts.registered.check_rand_calibration	27,639
realised_counts.registered.check_pairs_heldout	66,731
realised_counts.registered.check_gold_heldout	13,405
realised_counts.registered.check_rand_heldout	53,326
realised_counts.registered.heldout_R_hist.1	64
realised_counts.registered.heldout_R_hist.3	102
realised_counts.registered.heldout_R_hist.4	13,239
realised_counts.checked	yes
frame_comparison.frame_rows	24,044
frame_comparison.frame_pools	94
frame_comparison.dev_rows	24,044
frame_comparison.dev_pools	94
frame_comparison.equal	yes
split.recomputed_equals_job	yes
split.calibration_clusters	223
bootstrap.n_boot	2,000
bootstrap.seed	0
bootstrap.clusters	644
bootstrap.universe	421
rule.S1.ece_max	0.0300
rule.S1.ece_p975_max	0.0500
rule.S2.ece_max	0.0500
rule.S3.ece_max	0.0500
rule.S4.share_min	0.9900
rule.S4.prob_tol	0.0100
rule.S4.ece_diff_max	0.0100
rule.S5.ece_max	0.0300
rule.S5.ece_p975_max	0.0500
rule.S6.ece_max	0.0300
rule.S6.ece_p975_max	0.0500
rule.S6.band_ece_max	0.0500
rule.C1.ece_max	0.0300
rule.C1.ece_p975_max	0.0500
rule.C2.ece_max	0.0500
rule.C3.ba_min	0.8500
rule.C3.ba_p025_min	0.8000
rule.C3.threshold	0.5000
rule.C4.ece_max	0.0500
rule.C5.share_min	0.9900
rule.C5.prob_tol	0.0100
rule.C5.ece_diff_max	0.0100
calibration_file.written	yes
calibration_file.doc_without_dev_record.fusion.w	60.0
calibration_file.doc_without_dev_record.fusion.k	5
calibration_file.doc_without_dev_record.scores.temperature	2.4062
calibration_file.doc_without_dev_record.scores.remainder.gamma	-1.5022
calibration_file.doc_without_dev_record.scores.remainder.eta	0.1043
calibration_file.doc_without_dev_record.scores.remainder.kappa	0.8009
calibration_file.doc_without_dev_record.scores.k	5
calibration_file.file_sha256	n/a
seconds	126.2000

scores

path	value
scores.fit.ok	yes
scores.fit.T	2.4062
scores.fit.gamma	-1.5022
scores.fit.eta	0.1043
scores.fit.kappa	0.8009
scores.fit.beta	0.4156
scores.fit.converged	yes
scores.fit.iterations	8
scores.fit.nll	4575.3642
scores.fit.reason	n/a
scores.fit.rows	7,081
scores.fit.rows_over_5	7,011
scores.fit.rows_over_5_read	6,540
scores.fit.rows_over_5_not_read	471
scores.fit.min_rows	20
scores.fit.se.beta	0.0073
scores.fit.se.gamma	0.3349
scores.fit.se.eta	0.0148
scores.fit.se.kappa	0.0922
scores.flags.sums_ok	yes
scores.flags.remainder_zero_ok	yes
scores.flags.top_equals_frame	yes
scores.metrics.rows	13,405
scores.metrics.ece	0.0190
scores.metrics.ece_p975	0.0349
scores.metrics.ece_interval	[0.0121, 0.0349]
scores.metrics.cand_ece	0.0054
scores.metrics.cand_ece_p975	0.0103
scores.metrics.cand_ece_interval	[0.0038, 0.0103]
scores.metrics.rem_ece	0.0055
scores.metrics.rem_ece_p975	0.0178
scores.metrics.rem_ece_bands.6 to 20	0.0080
scores.metrics.rem_ece_bands.21 to 40	0.0122
scores.metrics.rem_ece_bands.41 or more	0.0130
scores.metrics.remainder_info.recall_at_5	0.9282
scores.metrics.remainder_info.over_5.rows	13,341
scores.metrics.remainder_info.over_5.not_read_share	0.0721
scores.metrics.remainder_info.over_5.mean_remainder	0.0750
scores.metrics.remainder_info.over_5.remainder_ece	0.0055
scores.metrics.remainder_info.by_size.6 to 20.rows	2,823
scores.metrics.remainder_info.by_size.6 to 20.not_read_share	0.0450
scores.metrics.remainder_info.by_size.6 to 20.mean_remainder	0.0374
scores.metrics.remainder_info.by_size.6 to 20.remainder_ece	0.0080
scores.metrics.remainder_info.by_size.21 to 40.rows	4,234
scores.metrics.remainder_info.by_size.21 to 40.not_read_share	0.0513
scores.metrics.remainder_info.by_size.21 to 40.mean_remainder	0.0509
scores.metrics.remainder_info.by_size.21 to 40.remainder_ece	0.0122
scores.metrics.remainder_info.by_size.41 or more.rows	6,284
scores.metrics.remainder_info.by_size.41 or more.not_read_share	0.0983
scores.metrics.remainder_info.by_size.41 or more.mean_remainder	0.1082
scores.metrics.remainder_info.by_size.41 or more.remainder_ece	0.0130
scores.metrics.rem_ece_band_intervals.6 to 20	[0.0038, 0.0343]
scores.metrics.rem_ece_band_intervals.21 to 40	[0.0060, 0.0245]
scores.metrics.rem_ece_band_intervals.41 or more	[0.0113, 0.0350]
scores.metrics.rem_ece_interval	[0.0054, 0.0178]
scores.metrics.recall_at_5	0.9282
scores.metrics.recall_at_5_interval	[0.9122, 0.9418]
scores.metrics.accuracy	0.7667
scores.metrics.accuracy_interval	[0.7415, 0.7915]
scores.metrics.mean_top	0.7522
scores.metrics.mean_top_interval	[0.7356, 0.7692]
scores.metrics.mean_top_minus_accuracy	-0.0145
scores.metrics.mean_top_minus_accuracy_interval	[-0.0313, 0.0032]
scores.metrics.brier	0.3204
scores.metrics.brier_interval	[0.2898, 0.3525]
scores.clause_inputs.fit_ok	yes
scores.clause_inputs.top_equals_frame	yes
scores.clause_inputs.sums_ok	yes
scores.clause_inputs.remainder_zero_ok	yes
scores.clause_inputs.ece	0.0190
scores.clause_inputs.ece_p975	0.0349
scores.clause_inputs.ece_EN	0.0164
scores.clause_inputs.ece_LTPL	0.0258
scores.clause_inputs.ece_EN_unseen	0.0324
scores.clause_inputs.ece_LTPL_unseen	0.0427
scores.clause_inputs.consistency.auto.share_within_tol	0.9994
scores.clause_inputs.consistency.auto.ece_diff	-0.0002
scores.clause_inputs.consistency.eager.share_within_tol	0.9991
scores.clause_inputs.consistency.eager.ece_diff	-0.0002
scores.clause_inputs.cand_ece	0.0054
scores.clause_inputs.cand_ece_p975	0.0103
scores.clause_inputs.rem_ece	0.0055
scores.clause_inputs.rem_ece_p975	0.0178
scores.clause_inputs.rem_ece_bands.6 to 20	0.0080
scores.clause_inputs.rem_ece_bands.21 to 40	0.0122
scores.clause_inputs.rem_ece_bands.41 or more	0.0130
scores.clauses.S0	yes
scores.clauses.S1	yes
scores.clauses.S2	yes
scores.clauses.S3	yes
scores.clauses.S4	yes
scores.clauses.S5	yes
scores.clauses.S6	yes
scores.ships	yes

scores.metrics.reliability.top_label	weight	count	mean_probability	outcome_rate
[0/15, 1/15)	0.0000	0	n/a	n/a
[1/15, 2/15)	4.0000	4	0.1215	0.0000
[2/15, 3/15)	163.0	163	0.1751	0.1840
[3/15, 4/15)	319.0	319	0.2370	0.2696
[4/15, 5/15)	565.0	565	0.3006	0.3097
[5/15, 6/15)	638.0	638	0.3676	0.4154
[6/15, 7/15)	678.0	678	0.4339	0.4469
[7/15, 8/15)	756.0	756	0.5005	0.5582
[8/15, 9/15)	777.0	777	0.5659	0.6049
[9/15, 10/15)	707.0	707	0.6336	0.6719
[10/15, 11/15)	742.0	742	0.7005	0.7278
[11/15, 12/15)	792.0	792	0.7674	0.7929
[12/15, 13/15)	907.0	907	0.8353	0.8467
[13/15, 14/15)	1,332.0	1,332	0.9039	0.9159
[14/15, 15/15]	5,025.0	5,025	0.9802	0.9743

scores.metrics.reliability.candidate	weight	count	mean_probability	outcome_rate
[0/15, 1/15)	8487.3500	42,394	0.0126	0.0109
[1/15, 2/15)	1126.7500	5,630	0.0952	0.0861
[2/15, 3/15)	565.5000	2,827	0.1648	0.1620
[3/15, 4/15)	350.4000	1,752	0.2315	0.2420
[4/15, 5/15)	277.9500	1,389	0.2985	0.3004
[5/15, 6/15)	206.6500	1,033	0.3656	0.3745
[6/15, 7/15)	174.6000	873	0.4325	0.4341
[7/15, 8/15)	156.2500	781	0.4997	0.5546
[8/15, 9/15)	155.4000	777	0.5659	0.6049
[9/15, 10/15)	141.4500	707	0.6336	0.6720
[10/15, 11/15)	148.5500	742	0.7005	0.7274
[11/15, 12/15)	158.4500	792	0.7674	0.7930
[12/15, 13/15)	181.5500	907	0.8353	0.8469
[13/15, 14/15)	267.1000	1,332	0.9039	0.9161
[14/15, 15/15]	1007.0500	5,025	0.9802	0.9744

scores.metrics.reliability.remainder_over_5	weight	count	mean_probability	outcome_rate
[0/15, 1/15)	8,655.0	8,655	0.0184	0.0179
[1/15, 2/15)	2,001.0	2,001	0.0968	0.1014
[2/15, 3/15)	1,146.0	1,146	0.1639	0.1483
[3/15, 4/15)	674.0	674	0.2306	0.2418
[4/15, 5/15)	405.0	405	0.2998	0.3012
[5/15, 6/15)	258.0	258	0.3628	0.3023
[6/15, 7/15)	159.0	159	0.4307	0.3333
[7/15, 8/15)	43.0	43	0.4862	0.4186
[8/15, 9/15)	0.0000	0	n/a	n/a
[9/15, 10/15)	0.0000	0	n/a	n/a
[10/15, 11/15)	0.0000	0	n/a	n/a
[11/15, 12/15)	0.0000	0	n/a	n/a
[12/15, 13/15)	0.0000	0	n/a	n/a
[13/15, 14/15)	0.0000	0	n/a	n/a
[14/15, 15/15]	0.0000	0	n/a	n/a

scores.groups	rows	ece	ece_p975	ece_interval	cand_ece	cand_ece_p975	cand_ece_interval	rem_ece
EN	4,725	0.0164	0.0350	[0.0140, 0.0350]	0.0039	0.0097	[0.0041, 0.0097]	0.0064
LTPL	4,943	0.0258	0.0462	[0.0183, 0.0462]	0.0082	0.0145	[0.0056, 0.0145]	0.0110
EN_unseen	2,569	0.0324	0.0524	[0.0264, 0.0524]	0.0111	0.0173	[0.0083, 0.0173]	0.0116
LTPL_unseen	3,050	0.0427	0.0661	[0.0296, 0.0661]	0.0145	0.0219	[0.0102, 0.0219]	0.0111
lt	2,530	0.0337	0.0559	[0.0256, 0.0559]	0.0083	0.0155	[0.0069, 0.0155]	0.0135
pl	2,413	0.0233	0.0474	[0.0205, 0.0474]	0.0096	0.0165	[0.0073, 0.0165]	0.0193
DESK	3,737	0.0252	0.0466	[0.0212, 0.0466]	0.0069	0.0138	[0.0059, 0.0138]	0.0097

scores.groups	rem_ece_p975	rem_ece_bands.6 to 20	rem_ece_bands.21 to 40	rem_ece_bands.41 or more	remainder_info.recall_at_5	remainder_info.over_5.rows	remainder_info.over_5.not_read_share	remainder_info.over_5.mean_remainder
EN	0.0170	0.0065	0.0126	0.0272	0.9509	4,709	0.0493	0.0494
LTPL	0.0252	0.0110	0.0175	0.0196	0.9124	4,918	0.0880	0.0942
EN_unseen	0.0238	0.00999	0.0257	0.0316	0.9533	2,553	0.0470	0.0568
LTPL_unseen	0.0284	0.0139	0.0260	0.0403	0.9213	3,025	0.0793	0.0892
lt	0.0304	0.0049	0.0303	0.0229	0.9182	2,516	0.0823	0.0947
pl	0.0356	0.0183	0.0270	0.0224	0.9063	2,402	0.0941	0.0936
DESK	0.0285	0.0134	0.0104	0.0165	0.9205	3,714	0.0800	0.0821

scores.groups	remainder_info.over_5.remainder_ece	remainder_info.by_size.6 to 20.rows	remainder_info.by_size.6 to 20.not_read_share	remainder_info.by_size.6 to 20.mean_remainder	remainder_info.by_size.6 to 20.remainder_ece	remainder_info.by_size.21 to 40.rows	remainder_info.by_size.21 to 40.not_read_share	remainder_info.by_size.21 to 40.mean_remainder
EN	0.0064	1,016	0.0246	0.0235	0.0065	2,308	0.0273	0.0345
LTPL	0.0110	904	0.0597	0.0488	0.0110	1,171	0.0863	0.0743
EN_unseen	0.0116	747	0.0335	0.0308	0.00999	925	0.0616	0.0627
LTPL_unseen	0.0111	791	0.0683	0.0543	0.0139	864	0.1019	0.0856
lt	0.0135	460	0.0522	0.0483	0.0049	602	0.0714	0.0732
pl	0.0193	444	0.0676	0.0492	0.0183	569	0.1019	0.0756
DESK	0.0097	903	0.0532	0.0415	0.0134	755	0.0702	0.0646

scores.groups	remainder_info.by_size.21 to 40.remainder_ece	remainder_info.by_size.41 or more.rows	remainder_info.by_size.41 or more.not_read_share	remainder_info.by_size.41 or more.mean_remainder	remainder_info.by_size.41 or more.remainder_ece	rem_ece_band_intervals.6 to 20	rem_ece_band_intervals.21 to 40	rem_ece_band_intervals.41 or more
EN	0.0126	1,385	0.1040	0.0931	0.0272	[0.0038, 0.0271]	[0.0084, 0.0206]	[0.0163, 0.0590]
LTPL	0.0175	2,843	0.0978	0.1168	0.0196	[0.0060, 0.05003]	[0.0098, 0.0423]	[0.0153, 0.0422]
EN_unseen	0.0257	881	0.0431	0.0726	0.0316	[0.0054, 0.0373]	[0.0142, 0.0437]	[0.0218, 0.0487]
LTPL_unseen	0.0260	1,370	0.0715	0.1116	0.0403	[0.0067, 0.0596]	[0.0113, 0.0590]	[0.0254, 0.0619]
lt	0.0303	1,454	0.0963	0.1184	0.0229	[0.0059, 0.0433]	[0.0159, 0.0538]	[0.0156, 0.0471]
pl	0.0270	1,389	0.0994	0.1152	0.0224	[0.0082, 0.0647]	[0.0154, 0.0616]	[0.0195, 0.0451]
DESK	0.0104	2,056	0.0953	0.1065	0.0165	[0.0068, 0.0505]	[0.0076, 0.0444]	[0.0129, 0.0434]

scores.groups	rem_ece_interval	recall_at_5	recall_at_5_interval	accuracy	accuracy_interval	mean_top	mean_top_interval	mean_top_minus_accuracy
EN	[0.0054, 0.0170]	0.9509	[0.9368, 0.9625]	0.8197	[0.7931, 0.8436]	0.8132	[0.7951, 0.8299]	-0.0065
LTPL	[0.0087, 0.0252]	0.9124	[0.8942, 0.9285]	0.7271	[0.6959, 0.7564]	0.7045	[0.6857, 0.7236]	-0.0226
EN_unseen	[0.0081, 0.0238]	0.9533	[0.9393, 0.9656]	0.8069	[0.7791, 0.8319]	0.7767	[0.7589, 0.7940]	-0.0302
LTPL_unseen	[0.0098, 0.0284]	0.9213	[0.9031, 0.9377]	0.7364	[0.7034, 0.7676]	0.6938	[0.6743, 0.7132]	-0.0426
lt	[0.0105, 0.0304]	0.9182	[0.8999, 0.9348]	0.7281	[0.6962, 0.7580]	0.7033	[0.6834, 0.7231]	-0.0248
pl	[0.0133, 0.0356]	0.9063	[0.8866, 0.9240]	0.7261	[0.6941, 0.7557]	0.7057	[0.6875, 0.7251]	-0.0204
DESK	[0.0079, 0.0285]	0.9205	[0.8989, 0.9389]	0.7522	[0.7186, 0.7853]	0.7383	[0.7168, 0.7609]	-0.0139

scores.groups	mean_top_minus_accuracy_interval	brier	brier_interval
EN	[-0.0233, 0.0118]	0.2512	[0.2210, 0.2855]
LTPL	[-0.0430, -0.0017]	0.3702	[0.3362, 0.4066]
EN_unseen	[-0.0491, -0.0099]	0.2710	[0.2415, 0.3052]
LTPL_unseen	[-0.0648, -0.0191]	0.3674	[0.3328, 0.4064]
lt	[-0.0470, -0.0024]	0.3690	[0.3329, 0.4063]
pl	[-0.0425, 0.0020]	0.3715	[0.3357, 0.4087]
DESK	[-0.0374, 0.0106]	0.3419	[0.3001, 0.3840]

scores.consistency	rows	rows_within_tol	share_within_tol	ece_library	ece_scorer	ece_diff	raw_fused_p99	raw_pairs
auto	3,236	3,234	0.9994	0.0267	0.0269	-0.0002	0.0110	16,180
eager	3,236	3,233	0.9991	0.0267	0.0269	-0.0002	0.0097	16,180

check

path	value
check.fit.ok	yes
check.fit.converged	yes
check.fit.a	0.2744
check.fit.b	-7.8925
check.fit.iterations	7
check.fit.nll	1991.6483
check.fit.reason	n/a
check.fit.pairs	34,720
check.fit.weight_pos	3540.5000
check.fit.weight_neg	3540.5000
check.fit.se.a	0.0063
check.fit.se.b	0.1834
check.metrics.pairs	66,731
check.metrics.ece	0.0141
check.metrics.ba	0.8730
check.metrics.tyr	0.8533
check.metrics.tnr	0.8926
check.metrics.brier	0.0926
check.metrics.auc	0.9432
check.metrics.right_yes_0.5	0.8882
check.metrics.right_yes_0.2	0.6652
check.metrics.right_yes_0.05	0.2949
check.metrics.ece_p975_twoway	0.03001
check.metrics.ba_p025_twoway	0.8557
check.metrics.ece_interval_twoway	[0.0082, 0.03001]
check.metrics.ba_interval_twoway	[0.8557, 0.8877]
check.metrics.tyr_interval_twoway	[0.8323, 0.8734]
check.metrics.tnr_interval_twoway	[0.8629, 0.9149]
check.metrics.brier_interval_twoway	[0.0827, 0.1043]
check.metrics.auc_interval_twoway	[0.9304, 0.9541]
check.metrics.right_yes_0.5_interval_twoway	[0.8621, 0.9094]
check.metrics.right_yes_0.2_interval_twoway	[0.6098, 0.7152]
check.metrics.right_yes_0.05_interval_twoway	[0.2476, 0.3458]
check.clause_inputs.converged	yes
check.clause_inputs.a	0.2744
check.clause_inputs.ece	0.0141
check.clause_inputs.ece_p975_twoway	0.03001
check.clause_inputs.ece_EN	0.0261
check.clause_inputs.ece_LTPL	0.03005
check.clause_inputs.ba	0.8730
check.clause_inputs.ba_p025_twoway	0.8557
check.clause_inputs.ece_EN_unseen	0.0566
check.clause_inputs.ece_LTPL_unseen	0.0149
check.clause_inputs.consistency.auto.share_within_tol	1.0000
check.clause_inputs.consistency.auto.ece_diff	-0.000001
check.clause_inputs.consistency.eager.share_within_tol	1.0000
check.clause_inputs.consistency.eager.ece_diff	0.0000001
check.clauses.C0	yes
check.clauses.C1	yes
check.clauses.C2	yes
check.clauses.C3	yes
check.clauses.C4	no
check.clauses.C5	yes
check.ships	no

check.metrics.reliability	weight	count	mean_probability	outcome_rate
[0/15, 1/15)	3507.5417	27,210	0.0251	0.0356
[1/15, 2/15)	1137.5417	8,311	0.0963	0.1134
[2/15, 3/15)	670.6250	4,620	0.1645	0.1842
[3/15, 4/15)	466.6667	3,016	0.2318	0.2561
[4/15, 5/15)	388.8750	2,313	0.2988	0.3369
[5/15, 6/15)	329.1250	1,865	0.3657	0.3844
[6/15, 7/15)	310.7500	1,601	0.4320	0.4714
[7/15, 8/15)	306.2500	1,451	0.4999	0.5388
[8/15, 9/15)	316.8750	1,377	0.5677	0.6059
[9/15, 10/15)	317.2500	1,272	0.6332	0.6619
[10/15, 11/15)	356.5000	1,298	0.7002	0.7209
[11/15, 12/15)	423.8750	1,414	0.7680	0.7679
[12/15, 13/15)	579.2500	1,604	0.8349	0.8632
[13/15, 14/15)	884.7500	2,200	0.9039	0.9138
[14/15, 15/15]	3409.1250	7,179	0.9780	0.9808

check.groups	pairs	ece	ba	tyr	tnr	brier	auc	right_yes_0.5
EN	23,475	0.0261	0.8929	0.9092	0.8765	0.0780	0.9590	0.8804
LTPL	24,640	0.03005	0.8589	0.8252	0.8925	0.1031	0.9325	0.8848
EN_unseen	12,797	0.0566	0.8672	0.9155	0.8188	0.0947	0.9484	0.8348
LTPL_unseen	15,175	0.0149	0.8572	0.8626	0.8517	0.1012	0.9358	0.8533
lt	12,608	0.0256	0.8591	0.8304	0.8878	0.1033	0.9320	0.8810
pl	12,032	0.0347	0.8586	0.8197	0.8974	0.1029	0.9331	0.8888
DESK	18,616	0.0438	0.8665	0.8199	0.9131	0.0971	0.9417	0.9042

check.groups	right_yes_0.2	right_yes_0.05	ece_p975_twoway	ba_p025_twoway	ece_interval_twoway	ba_interval_twoway	tyr_interval_twoway	tnr_interval_twoway
EN	0.6480	0.2793	0.0430	0.8750	[0.0154, 0.0430]	[0.8750, 0.9078]	[0.8891, 0.9265]	[0.8440, 0.9028]
LTPL	0.6575	0.2878	0.0485	0.8407	[0.0159, 0.0485]	[0.8407, 0.8752]	[0.8001, 0.8491]	[0.8597, 0.9170]
EN_unseen	0.5581	0.2101	0.0790	0.8443	[0.0375, 0.0790]	[0.8443, 0.8868]	[0.8966, 0.9326]	[0.7760, 0.8565]
LTPL_unseen	0.5926	0.2344	0.0379	0.8354	[0.0140, 0.0379]	[0.8354, 0.8767]	[0.8408, 0.8828]	[0.8111, 0.8843]
lt	0.6493	0.2804	0.0450	0.8406	[0.0161, 0.0450]	[0.8406, 0.8760]	[0.8042, 0.8546]	[0.8534, 0.9129]
pl	0.6664	0.2961	0.0536	0.8400	[0.0211, 0.0536]	[0.8400, 0.8753]	[0.7934, 0.8458]	[0.8661, 0.9225]
DESK	0.7023	0.3318	0.0655	0.8446	[0.0250, 0.0655]	[0.8446, 0.8859]	[0.7875, 0.8503]	[0.8799, 0.9392]

check.groups	brier_interval_twoway	auc_interval_twoway	right_yes_0.5_interval_twoway	right_yes_0.2_interval_twoway	right_yes_0.05_interval_twoway
EN	[0.0681, 0.0900]	[0.9480, 0.9680]	[0.8544, 0.9032]	[0.5946, 0.7000]	[0.2359, 0.3294]
LTPL	[0.0922, 0.1152]	[0.9170, 0.9458]	[0.8563, 0.9084]	[0.5983, 0.7126]	[0.2387, 0.3429]
EN_unseen	[0.0818, 0.1097]	[0.9346, 0.9605]	[0.8044, 0.8648]	[0.5069, 0.6152]	[0.1779, 0.2518]
LTPL_unseen	[0.0890, 0.1157]	[0.9182, 0.9498]	[0.8210, 0.8815]	[0.5342, 0.6504]	[0.1945, 0.2814]
lt	[0.0922, 0.1157]	[0.9163, 0.9453]	[0.8506, 0.9048]	[0.5874, 0.7037]	[0.2306, 0.3333]
pl	[0.0918, 0.1152]	[0.9171, 0.9469]	[0.8613, 0.9132]	[0.6083, 0.7245]	[0.2464, 0.3564]
DESK	[0.0841, 0.1121]	[0.9237, 0.9565]	[0.8728, 0.9308]	[0.6317, 0.7708]	[0.2653, 0.4145]

check.consistency	pairs	pairs_with_library_score	pairs_within_tol	share_within_tol	ece_library	ece_scorer	ece_diff	raw_fused_p99
auto	16,180	16,180	16,180	1.0000	0.1004	0.1004	-0.000001	0.0126
eager	16,180	16,180	16,180	1.0000	0.1004	0.1004	0.0000001	0.0125

information

path	value
information.split_counts.kept.rows	20,486
information.split_counts.kept.clusters	644
information.split_counts.kept.messages	8,530
information.split_counts.kept.pools	85
information.split_counts.by_side.calibration.rows	7,081
information.split_counts.by_side.calibration.clusters	223
information.split_counts.by_side.calibration.messages	3,023
information.split_counts.by_side.calibration.pools	85
information.split_counts.by_side.heldout.rows	13,405
information.split_counts.by_side.heldout.clusters	421
information.split_counts.by_side.heldout.messages	5,507
information.split_counts.by_side.heldout.pools	85
information.split_counts.check_pairs.calibration.pairs	34,720
information.split_counts.check_pairs.calibration.gold	7,081
information.split_counts.check_pairs.calibration.rand	27,639
information.split_counts.check_pairs.calibration.R_hist.1	70
information.split_counts.check_pairs.calibration.R_hist.2	117
information.split_counts.check_pairs.calibration.R_hist.3	241
information.split_counts.check_pairs.calibration.R_hist.4	6,653
information.split_counts.check_pairs.calibration.negative_intents	223
information.split_counts.check_pairs.heldout.pairs	66,731
information.split_counts.check_pairs.heldout.gold	13,405
information.split_counts.check_pairs.heldout.rand	53,326
information.split_counts.check_pairs.heldout.R_hist.1	64
information.split_counts.check_pairs.heldout.R_hist.3	102
information.split_counts.check_pairs.heldout.R_hist.4	13,239
information.split_counts.check_pairs.heldout.negative_intents	421
information.split_counts.labels.base_pools	26
information.split_counts.labels.labels	644
information.split_counts.labels.calibration	223
information.split_counts.labels.heldout	421
information.split_counts.strict_rows.calibration	2,748
information.split_counts.strict_rows.heldout	5,880
information.split_counts.text_overlap.texts_in_several_base_pools	33
information.split_counts.text_overlap.rows_with_a_text_of_several_base_pools	66
information.split_counts.text_overlap.base_pool_sets.dstc11_banking_guard + dstc11_finance_guard	31
information.split_counts.text_overlap.base_pool_sets.ind_dev_unseen_telecom_operator + synth_dev_unseen_yoga_studio	2
information.split_counts.text_overlap.texts_in_several_clusters	34
information.split_counts.text_overlap.texts_on_both_sides	0
information.split_counts.p_dev.pools	94
information.split_counts.p_dev.rows	24,044
information.split_counts.p_dev.decision_rows	23,744
information.split_counts.p_dev.rows_uncapped	33,341
information.split_counts.p_dev.equals_stored_frame_count	yes
information.population.excluded.clinc_dev (English): P's checkpoint-selection pool, left out as NW-P's EN_DEC leaves it out.rows	1,800
information.population.excluded.clinc_dev (English): P's checkpoint-selection pool, left out as NW-P's EN_DEC leaves it out.decision_rows	1,716
information.population.excluded.ind_dev_seen_industry: training labels by construction (entry of 2026-10-12 11:02).rows	150
information.population.excluded.ind_dev_seen_industry: training labels by construction (entry of 2026-10-12 11:02).decision_rows	149
information.population.excluded.mixed_dev: 2 labels (group EXCLUDED).rows	60
information.population.excluded.mixed_dev: 2 labels (group EXCLUDED).decision_rows	58
information.population.excluded.nlupp_hotels_dev: nested labels ('not this intent' ill defined).rows	1,046
information.population.excluded.nlupp_hotels_dev: nested labels ('not this intent' ill defined).decision_rows	1,045
information.population.excluded.synth_dev_seen_domain_v12: outside the DEV freeze (D8).rows	290
information.population.excluded.synth_dev_seen_domain_v12: outside the DEV freeze (D8).decision_rows	290
information.population.near_test_removed.rows	212
information.population.near_test_removed.by_lang.en	212
information.population.near_test_removed.by_lang.de	0
information.population.near_test_removed.by_lang.lt	0
information.population.near_test_removed.by_lang.pl	0
information.population.near_test_removed.by_lang.sk	0
information.consistency_rows	3,236
information.strict.scores.rows	5,880
information.strict.scores.ece	0.0315
information.strict.scores.ece_p975	0.0556
information.strict.scores.ece_interval	[0.0213, 0.0556]
information.strict.scores.cand_ece	0.0122
information.strict.scores.cand_ece_p975	0.0194
information.strict.scores.cand_ece_interval	[0.0085, 0.0194]
information.strict.scores.rem_ece	0.0127
information.strict.scores.rem_ece_p975	0.0384
information.strict.scores.rem_ece_bands.6 to 20	0.0278
information.strict.scores.rem_ece_bands.21 to 40	0.0311
information.strict.scores.rem_ece_bands.41 or more	0.0186
information.strict.scores.remainder_info.recall_at_5	0.9088
information.strict.scores.remainder_info.over_5.rows	5,845
information.strict.scores.remainder_info.over_5.not_read_share	0.0917
information.strict.scores.remainder_info.over_5.mean_remainder	0.0816
information.strict.scores.remainder_info.over_5.remainder_ece	0.0127
information.strict.scores.remainder_info.by_size.6 to 20.rows	1,295
information.strict.scores.remainder_info.by_size.6 to 20.not_read_share	0.0795
information.strict.scores.remainder_info.by_size.6 to 20.mean_remainder	0.0517
information.strict.scores.remainder_info.by_size.6 to 20.remainder_ece	0.0278
information.strict.scores.remainder_info.by_size.21 to 40.rows	1,581
information.strict.scores.remainder_info.by_size.21 to 40.not_read_share	0.0753
information.strict.scores.remainder_info.by_size.21 to 40.mean_remainder	0.0528
information.strict.scores.remainder_info.by_size.21 to 40.remainder_ece	0.0311
information.strict.scores.remainder_info.by_size.41 or more.rows	2,969
information.strict.scores.remainder_info.by_size.41 or more.not_read_share	0.1058
information.strict.scores.remainder_info.by_size.41 or more.mean_remainder	0.1099
information.strict.scores.remainder_info.by_size.41 or more.remainder_ece	0.0186
information.strict.scores.rem_ece_band_intervals.6 to 20	[0.0080, 0.0825]
information.strict.scores.rem_ece_band_intervals.21 to 40	[0.0123, 0.0631]
information.strict.scores.rem_ece_band_intervals.41 or more	[0.0146, 0.0469]
information.strict.scores.rem_ece_interval	[0.0085, 0.0384]
information.strict.scores.recall_at_5	0.9088
information.strict.scores.recall_at_5_interval	[0.8791, 0.9341]
information.strict.scores.accuracy	0.7646
information.strict.scores.accuracy_interval	[0.7249, 0.8022]
information.strict.scores.mean_top	0.7406
information.strict.scores.mean_top_interval	[0.7142, 0.7673]
information.strict.scores.mean_top_minus_accuracy	-0.0240
information.strict.scores.mean_top_minus_accuracy_interval	[-0.0493, 0.0019]
information.strict.scores.brier	0.3180
information.strict.scores.brier_interval	[0.2758, 0.3637]
information.strict.check.pairs	29,236
information.strict.check.ece	0.0275
information.strict.check.ba	0.8509
information.strict.check.tyr	0.8173
information.strict.check.tnr	0.8844
information.strict.check.brier	0.1065
information.strict.check.auc	0.9293
information.strict.check.right_yes_0.5	0.8761
information.strict.check.right_yes_0.2	0.6387
information.strict.check.right_yes_0.05	0.2712
information.strict.check.ece_p975_twoway	0.0524
information.strict.check.ba_p025_twoway	0.8246
information.strict.check.ece_interval_twoway	[0.0162, 0.0524]
information.strict.check.ba_interval_twoway	[0.8246, 0.8735]
information.strict.check.tyr_interval_twoway	[0.7780, 0.8518]
information.strict.check.tnr_interval_twoway	[0.8441, 0.9164]
information.strict.check.brier_interval_twoway	[0.0913, 0.1239]
information.strict.check.auc_interval_twoway	[0.9083, 0.9467]
information.strict.check.right_yes_0.5_interval_twoway	[0.8396, 0.9072]
information.strict.check.right_yes_0.2_interval_twoway	[0.5669, 0.7095]
information.strict.check.right_yes_0.05_interval_twoway	[0.2160, 0.3396]
information.domain_seen_by_P.scores.rows	5,940
information.domain_seen_by_P.scores.ece	0.0194
information.domain_seen_by_P.scores.ece_p975	0.0485
information.domain_seen_by_P.scores.ece_interval	[0.0178, 0.0485]
information.domain_seen_by_P.scores.cand_ece	0.0071
information.domain_seen_by_P.scores.cand_ece_p975	0.0161
information.domain_seen_by_P.scores.cand_ece_interval	[0.0054, 0.0161]
information.domain_seen_by_P.scores.rem_ece	0.0104
information.domain_seen_by_P.scores.rem_ece_p975	0.0331
information.domain_seen_by_P.scores.rem_ece_bands.6 to 20	0.0058
information.domain_seen_by_P.scores.rem_ece_bands.21 to 40	0.0098
information.domain_seen_by_P.scores.rem_ece_bands.41 or more	0.0209
information.domain_seen_by_P.scores.remainder_info.recall_at_5	0.9195
information.domain_seen_by_P.scores.remainder_info.over_5.rows	5,940
information.domain_seen_by_P.scores.remainder_info.over_5.not_read_share	0.0805
information.domain_seen_by_P.scores.remainder_info.over_5.mean_remainder	0.0765
information.domain_seen_by_P.scores.remainder_info.over_5.remainder_ece	0.0104
information.domain_seen_by_P.scores.remainder_info.by_size.6 to 20.rows	498
information.domain_seen_by_P.scores.remainder_info.by_size.6 to 20.not_read_share	0.0000
information.domain_seen_by_P.scores.remainder_info.by_size.6 to 20.mean_remainder	0.0058
information.domain_seen_by_P.scores.remainder_info.by_size.6 to 20.remainder_ece	0.0058
information.domain_seen_by_P.scores.remainder_info.by_size.21 to 40.rows	2,006
information.domain_seen_by_P.scores.remainder_info.by_size.21 to 40.not_read_share	0.0140
information.domain_seen_by_P.scores.remainder_info.by_size.21 to 40.mean_remainder	0.0238
information.domain_seen_by_P.scores.remainder_info.by_size.21 to 40.remainder_ece	0.0098
information.domain_seen_by_P.scores.remainder_info.by_size.41 or more.rows	3,436
information.domain_seen_by_P.scores.remainder_info.by_size.41 or more.not_read_share	0.1310
information.domain_seen_by_P.scores.remainder_info.by_size.41 or more.mean_remainder	0.1175
information.domain_seen_by_P.scores.remainder_info.by_size.41 or more.remainder_ece	0.0209
information.domain_seen_by_P.scores.rem_ece_band_intervals.6 to 20	[0.0036, 0.0080]
information.domain_seen_by_P.scores.rem_ece_band_intervals.21 to 40	[0.0050, 0.0204]
information.domain_seen_by_P.scores.rem_ece_band_intervals.41 or more	[0.0142, 0.0628]
information.domain_seen_by_P.scores.rem_ece_interval	[0.0081, 0.0331]
information.domain_seen_by_P.scores.recall_at_5	0.9195
information.domain_seen_by_P.scores.recall_at_5_interval	[0.8882, 0.9460]
information.domain_seen_by_P.scores.accuracy	0.7638
information.domain_seen_by_P.scores.accuracy_interval	[0.7171, 0.8087]
information.domain_seen_by_P.scores.mean_top	0.7782
information.domain_seen_by_P.scores.mean_top_interval	[0.7480, 0.8072]
information.domain_seen_by_P.scores.mean_top_minus_accuracy	0.0144
information.domain_seen_by_P.scores.mean_top_minus_accuracy_interval	[-0.0152, 0.0440]
information.domain_seen_by_P.scores.brier	0.3170
information.domain_seen_by_P.scores.brier_interval	[0.2614, 0.3752]
information.domain_seen_by_P.check.pairs	29,598
information.domain_seen_by_P.check.ece	0.0664
information.domain_seen_by_P.check.ba	0.8853
information.domain_seen_by_P.check.tyr	0.8160
information.domain_seen_by_P.check.tnr	0.9546
information.domain_seen_by_P.check.brier	0.0861
information.domain_seen_by_P.check.auc	0.9563
information.domain_seen_by_P.check.right_yes_0.5	0.9473
information.domain_seen_by_P.check.right_yes_0.2	0.8181
information.domain_seen_by_P.check.right_yes_0.05	0.4863
information.domain_seen_by_P.check.ece_p975_twoway	0.0923
information.domain_seen_by_P.check.ba_p025_twoway	0.8613
information.domain_seen_by_P.check.ece_interval_twoway	[0.0433, 0.0923]
information.domain_seen_by_P.check.ba_interval_twoway	[0.8613, 0.9074]
information.domain_seen_by_P.check.tyr_interval_twoway	[0.7741, 0.8563]
information.domain_seen_by_P.check.tnr_interval_twoway	[0.9322, 0.9704]
information.domain_seen_by_P.check.brier_interval_twoway	[0.0709, 0.1024]
information.domain_seen_by_P.check.auc_interval_twoway	[0.9410, 0.9694]
information.domain_seen_by_P.check.right_yes_0.5_interval_twoway	[0.9240, 0.9649]
information.domain_seen_by_P.check.right_yes_0.2_interval_twoway	[0.7524, 0.8731]
information.domain_seen_by_P.check.right_yes_0.05_interval_twoway	[0.3901, 0.5915]
information.hard_negatives.pairs	54,518
information.hard_negatives.rows	13,405
information.hard_negatives.false_yes_rate	0.4198
information.hard_negatives.false_yes_rate_interval	[0.3915, 0.4493]
information.hard_negatives.rank2_confuser_false_yes_rate	0.6292
information.hard_negatives.rank2_confuser_interval	[0.5968, 0.6620]
information.check_name_only.pairs	66,731
information.check_name_only.ece	0.1214
information.check_name_only.ba	0.8357
information.check_name_only.tyr	0.9411
information.check_name_only.tnr	0.7303
information.check_name_only.brier	0.1212
information.check_name_only.auc	0.9399
information.check_name_only.right_yes_0.5	0.7772
information.check_name_only.right_yes_0.2	0.4659
information.check_name_only.right_yes_0.05	0.1551
information.check_name_only.ece_p975_twoway	0.1416
information.check_name_only.ba_p025_twoway	0.8156
information.check_name_only.ece_interval_twoway	[0.1047, 0.1416]
information.check_name_only.ba_interval_twoway	[0.8156, 0.8526]
information.check_name_only.tyr_interval_twoway	[0.9250, 0.9553]
information.check_name_only.tnr_interval_twoway	[0.6897, 0.7652]
information.check_name_only.brier_interval_twoway	[0.1094, 0.1360]
information.check_name_only.auc_interval_twoway	[0.9275, 0.9501]
information.check_name_only.right_yes_0.5_interval_twoway	[0.7527, 0.7998]
information.check_name_only.right_yes_0.2_interval_twoway	[0.4321, 0.4996]
information.check_name_only.right_yes_0.05_interval_twoway	[0.1381, 0.1737]
information.english_clinc_dev.scores.rows	1,154
information.english_clinc_dev.scores.ece	0.0302
information.english_clinc_dev.scores.ece_p975	0.0794
information.english_clinc_dev.scores.ece_interval	[0.0259, 0.0794]
information.english_clinc_dev.scores.cand_ece	0.0103
information.english_clinc_dev.scores.cand_ece_p975	0.0267
information.english_clinc_dev.scores.cand_ece_interval	[0.0099, 0.0267]
information.english_clinc_dev.scores.rem_ece	0.0415
information.english_clinc_dev.scores.rem_ece_p975	0.0616
information.english_clinc_dev.scores.rem_ece_bands.6 to 20	n/a
information.english_clinc_dev.scores.rem_ece_bands.21 to 40	n/a
information.english_clinc_dev.scores.rem_ece_bands.41 or more	0.0415
information.english_clinc_dev.scores.remainder_info.recall_at_5	0.9818
information.english_clinc_dev.scores.remainder_info.over_5.rows	1,154
information.english_clinc_dev.scores.remainder_info.over_5.not_read_share	0.0182
information.english_clinc_dev.scores.remainder_info.over_5.mean_remainder	0.0597
information.english_clinc_dev.scores.remainder_info.over_5.remainder_ece	0.0415
information.english_clinc_dev.scores.remainder_info.by_size.6 to 20.rows	0
information.english_clinc_dev.scores.remainder_info.by_size.6 to 20.not_read_share	n/a
information.english_clinc_dev.scores.remainder_info.by_size.6 to 20.mean_remainder	n/a
information.english_clinc_dev.scores.remainder_info.by_size.6 to 20.remainder_ece	n/a
information.english_clinc_dev.scores.remainder_info.by_size.21 to 40.rows	0
information.english_clinc_dev.scores.remainder_info.by_size.21 to 40.not_read_share	n/a
information.english_clinc_dev.scores.remainder_info.by_size.21 to 40.mean_remainder	n/a
information.english_clinc_dev.scores.remainder_info.by_size.21 to 40.remainder_ece	n/a
information.english_clinc_dev.scores.remainder_info.by_size.41 or more.rows	1,154
information.english_clinc_dev.scores.remainder_info.by_size.41 or more.not_read_share	0.0182
information.english_clinc_dev.scores.remainder_info.by_size.41 or more.mean_remainder	0.0597
information.english_clinc_dev.scores.remainder_info.by_size.41 or more.remainder_ece	0.0415
information.english_clinc_dev.scores.rem_ece_band_intervals.6 to 20	n/a
information.english_clinc_dev.scores.rem_ece_band_intervals.21 to 40	n/a
information.english_clinc_dev.scores.rem_ece_band_intervals.41 or more	[0.0283, 0.0616]
information.english_clinc_dev.scores.rem_ece_interval	[0.0283, 0.0616]
information.english_clinc_dev.scores.recall_at_5	0.9818
information.english_clinc_dev.scores.recall_at_5_interval	[0.9552, 0.9985]
information.english_clinc_dev.scores.accuracy	0.8388
information.english_clinc_dev.scores.accuracy_interval	[0.7622, 0.9045]
information.english_clinc_dev.scores.mean_top	0.8139
information.english_clinc_dev.scores.mean_top_interval	[0.7630, 0.8616]
information.english_clinc_dev.scores.mean_top_minus_accuracy	-0.0249
information.english_clinc_dev.scores.mean_top_minus_accuracy_interval	[-0.0753, 0.0333]
information.english_clinc_dev.scores.brier	0.2369
information.english_clinc_dev.scores.brier_interval	[0.1474, 0.3375]
information.english_clinc_dev.check.pairs	5,770
information.english_clinc_dev.check.ece	0.1051
information.english_clinc_dev.check.ba	0.9046
information.english_clinc_dev.check.tyr	0.8172
information.english_clinc_dev.check.tnr	0.9920
information.english_clinc_dev.check.brier	0.0722
information.english_clinc_dev.check.auc	0.9885
information.english_clinc_dev.check.right_yes_0.5	0.9903
information.english_clinc_dev.check.right_yes_0.2	0.9622
information.english_clinc_dev.check.right_yes_0.05	0.8429
information.english_clinc_dev.check.ece_p975_twoway	0.1531
information.english_clinc_dev.check.ba_p025_twoway	0.8507
information.english_clinc_dev.check.ece_interval_twoway	[0.0728, 0.1531]
information.english_clinc_dev.check.ba_interval_twoway	[0.8507, 0.9419]
information.english_clinc_dev.check.tyr_interval_twoway	[0.7244, 0.8957]
information.english_clinc_dev.check.tnr_interval_twoway	[0.9840, 0.9974]
information.english_clinc_dev.check.brier_interval_twoway	[0.0463, 0.1109]
information.english_clinc_dev.check.auc_interval_twoway	[0.9799, 0.9950]
information.english_clinc_dev.check.right_yes_0.5_interval_twoway	[0.9807, 0.9967]
information.english_clinc_dev.check.right_yes_0.2_interval_twoway	[0.9270, 0.9870]
information.english_clinc_dev.check.right_yes_0.05_interval_twoway	[0.7278, 0.9411]
information.fitted.scores.T	2.4062
information.fitted.scores.gamma	-1.5022
information.fitted.scores.eta	0.1043
information.fitted.scores.kappa	0.8009
information.fitted.scores.beta	0.4156
information.fitted.scores.se.beta	0.0073
information.fitted.scores.se.gamma	0.3349
information.fitted.scores.se.eta	0.0148
information.fitted.scores.se.kappa	0.0922
information.fitted.scores.iterations	8
information.fitted.scores.converged	yes
information.fitted.scores.rows	7,081
information.fitted.scores.rows_over_5	7,011
information.fitted.scores.rows_over_5_read	6,540
information.fitted.scores.rows_over_5_not_read	471
information.fitted.scores.reason	n/a
information.fitted.check.a	0.2744
information.fitted.check.b	-7.8925
information.fitted.check.se.a	0.0063
information.fitted.check.se.b	0.1834
information.fitted.check.iterations	7
information.fitted.check.converged	yes
information.fitted.check.pairs	34,720
information.fitted.check.reason	n/a

information.split_counts.by_group	calibration.rows	calibration.clusters	calibration.messages	calibration.pools	heldout.rows	heldout.clusters	heldout.messages	heldout.pools
EN	2,631	203	2,631	25	4,725	381	4,725	25
LTPL	2,542	168	1,366	34	4,943	321	2,619	34
EN_unseen	1,406	147	1,406	19	2,569	277	2,569	19
LTPL_unseen	1,605	112	851	26	3,050	213	1,601	26
lt	1,308	168	1,308	17	2,530	321	2,530	17
pl	1,234	168	1,234	17	2,413	321	2,413	17
DESK	1,908	123	1,009	26	3,737	234	1,982	26

information.split_counts.by_group	all.rows	all.clusters	all.messages	all.pools
EN	7,356	584	7,356	25
LTPL	7,485	489	3,985	34
EN_unseen	3,975	424	3,975	19
LTPL_unseen	4,655	325	2,452	26
lt	3,838	489	3,838	17
pl	3,647	489	3,647	17
DESK	5,645	357	2,991	26

information.split_counts.by_lang	calibration.rows	calibration.clusters	calibration.messages	calibration.pools	heldout.rows	heldout.clusters	heldout.messages	heldout.pools
de	960	123	960	13	1,876	234	1,876	13
en	2,631	203	2,631	25	4,725	381	4,725	25
lt	1,308	168	1,308	17	2,530	321	2,530	17
pl	1,234	168	1,234	17	2,413	321	2,413	17
sk	948	123	948	13	1,861	234	1,861	13

information.by_pool	scores.rows	scores.ece	scores.ece_p975	scores.ece_interval	scores.cand_ece	scores.cand_ece_p975	scores.cand_ece_interval	scores.rem_ece
abcd_dev	504	0.1080	0.1779	[0.0762, 0.1779]	0.0296	0.0502	[0.0212, 0.0502]	0.0883
abcd_dev@de	260	0.1551	0.2380	[0.1213, 0.2380]	0.0394	0.0675	[0.0294, 0.0675]	0.1000
abcd_dev@lt	265	0.0931	0.1968	[0.0759, 0.1968]	0.0356	0.0646	[0.0251, 0.0646]	0.1048
abcd_dev@pl	243	0.1138	0.2071	[0.0916, 0.2071]	0.0350	0.0622	[0.0282, 0.0622]	0.0956
abcd_dev@sk	253	0.0988	0.1931	[0.0818, 0.1931]	0.0226	0.0507	[0.0212, 0.0507]	0.0990
bitext_dev	508	0.0708	0.1584	[0.0359, 0.1584]	0.0203	0.0519	[0.0123, 0.0519]	0.0136
clinc_dev@de	464	0.0490	0.1086	[0.0414, 0.1086]	0.0143	0.0351	[0.0129, 0.0351]	0.0360
clinc_dev@lt	486	0.0580	0.1158	[0.0522, 0.1158]	0.0163	0.0339	[0.0157, 0.0339]	0.0525
clinc_dev@pl	479	0.05004	0.1031	[0.0459, 0.1031]	0.0121	0.0302	[0.0139, 0.0302]	0.0487
clinc_dev@sk	482	0.0321	0.0944	[0.0382, 0.0944]	0.0153	0.0344	[0.0142, 0.0344]	0.0558
dstc11_banking_guard	167	0.0399	0.0653	[0.0329, 0.0653]	0.0156	0.0248	[0.0125, 0.0248]	0.0047
dstc11_banking_guard@de	65	0.0789	0.1669	[0.0368, 0.1669]	0.0287	0.0544	[0.0134, 0.0544]	0.0069
dstc11_banking_guard@lt	62	0.0686	0.1396	[0.0444, 0.1396]	0.0274	0.0482	[0.0165, 0.0482]	0.0094
dstc11_banking_guard@pl	51	0.0932	0.1768	[0.0367, 0.1768]	0.0319	0.0659	[0.0132, 0.0659]	0.0103
dstc11_banking_guard@sk	51	0.0956	0.1901	[0.0341, 0.1901]	0.0267	0.0513	[0.0123, 0.0513]	0.0086
dstc11_finance_guard	498	0.0513	0.1067	[0.0319, 0.1067]	0.0155	0.0384	[0.0101, 0.0384]	0.0119
dstc11_finance_guard@de	165	0.0888	0.1687	[0.0564, 0.1687]	0.0194	0.0483	[0.0150, 0.0483]	0.0329
dstc11_finance_guard@lt	166	0.0463	0.1276	[0.0425, 0.1276]	0.0203	0.0462	[0.0172, 0.0462]	0.0190
dstc11_finance_guard@pl	141	0.0824	0.1729	[0.0575, 0.1729]	0.0292	0.0620	[0.0197, 0.0620]	0.0261
dstc11_finance_guard@sk	151	0.0831	0.1768	[0.0584, 0.1768]	0.0305	0.0646	[0.0197, 0.0646]	0.0292
dstc11_insurance	377	0.0418	0.1420	[0.0137, 0.1420]	0.0176	0.0590	[0.0049, 0.0590]	0.0096
hint3_dev	67	0.1526	n/a	[0.0375, 0.3446]	0.0550	n/a	[0.0177, 0.1112]	0.1078
ind_dev_unseen_electricity_grid_operator	147	0.1191	0.1956	[0.0821, 0.1956]	0.0391	0.0626	[0.0269, 0.0626]	0.0410
ind_dev_unseen_electricity_grid_operator@lt	101	0.1318	0.2341	[0.0975, 0.2341]	0.0378	0.0705	[0.0311, 0.0705]	0.0391
ind_dev_unseen_electricity_grid_operator@pl	95	0.1545	0.2725	[0.1253, 0.2725]	0.0411	0.0777	[0.0378, 0.0777]	0.0341
ind_dev_unseen_property_management_company	258	0.0515	0.1102	[0.0522, 0.1102]	0.0209	0.0380	[0.0176, 0.0380]	0.0196
ind_dev_unseen_property_management_company@lt	200	0.0691	0.1563	[0.0694, 0.1563]	0.0215	0.0481	[0.0213, 0.0481]	0.0622
ind_dev_unseen_property_management_company@pl	184	0.1007	0.1881	[0.0921, 0.1881]	0.0289	0.0548	[0.0266, 0.0548]	0.0316
ind_dev_unseen_tax_and_social_security_office	246	0.0809	0.1351	[0.0642, 0.1351]	0.0268	0.0440	[0.0223, 0.0440]	0.0437
ind_dev_unseen_tax_and_social_security_office@lt	197	0.1173	0.1972	[0.0905, 0.1972]	0.0316	0.0560	[0.0274, 0.0560]	0.0914
ind_dev_unseen_tax_and_social_security_office@pl	192	0.1361	0.1843	[0.1080, 0.1843]	0.0399	0.0567	[0.0367, 0.0567]	0.0599
ind_dev_unseen_telecom_operator	153	0.0666	0.1762	[0.0553, 0.1762]	0.0215	0.0610	[0.0196, 0.0610]	0.0272
ind_dev_unseen_telecom_operator@lt	118	0.0844	0.2026	[0.0579, 0.2026]	0.0280	0.0836	[0.0213, 0.0836]	0.0374
ind_dev_unseen_telecom_operator@pl	115	0.1117	0.2478	[0.0826, 0.2478]	0.0387	0.0915	[0.0297, 0.0915]	0.0642
sgd_unseen	102	0.0987	n/a	[0.0046, 0.3476]	0.0397	n/a	[0.0018, 0.1382]	0.0010
synth_dev_seen_domain	377	0.0432	0.0850	[0.0380, 0.0850]	0.0141	0.0284	[0.0108, 0.0284]	0.0453
synth_dev_seen_domain@de	299	0.0646	0.1112	[0.0501, 0.1112]	0.0208	0.0346	[0.0160, 0.0346]	0.0341
synth_dev_seen_domain@lt	306	0.0602	0.1060	[0.0538, 0.1060]	0.0188	0.0325	[0.0149, 0.0325]	0.0424
synth_dev_seen_domain@pl	291	0.0831	0.1343	[0.0681, 0.1343]	0.0213	0.0390	[0.0173, 0.0390]	0.0597
synth_dev_seen_domain@sk	298	0.0889	0.1286	[0.0717, 0.1286]	0.0232	0.0355	[0.0182, 0.0355]	0.0739
synth_dev_unseen_bookstore	16	0.0858	n/a	[0.0230, 0.1234]	0.0430	n/a	[0.0115, 0.0619]	n/a
synth_dev_unseen_bookstore@de	11	0.0359	n/a	[0.029997, 0.0430]	0.0179	n/a	[0.0150, 0.0215]	n/a
synth_dev_unseen_bookstore@lt	14	0.1125	n/a	[0.0569, 0.2161]	0.0558	n/a	[0.0325, 0.0902]	n/a
synth_dev_unseen_bookstore@pl	11	0.0723	n/a	[0.0298, 0.1078]	0.0362	n/a	[0.0149, 0.0539]	n/a
synth_dev_unseen_bookstore@sk	12	0.0566	n/a	[0.0489, 0.0643]	0.0283	n/a	[0.0245, 0.0321]	n/a
synth_dev_unseen_camera_store	45	0.1703	n/a	[0.0970, 0.3688]	0.0574	n/a	[0.0336, 0.1018]	0.0088
synth_dev_unseen_car_dealership_service_department	57	0.1353	n/a	[0.0826, 0.2213]	0.0471	n/a	[0.0423, 0.0859]	0.0152
synth_dev_unseen_car_wash_subscription	73	0.1077	0.2724	[0.0887, 0.2724]	0.0377	0.0881	[0.0313, 0.0881]	0.0305
synth_dev_unseen_dental_practice	148	0.1268	0.2127	[0.1043, 0.2127]	0.0307	0.0630	[0.0294, 0.0630]	0.0436
synth_dev_unseen_dental_practice@de	115	0.0939	0.2241	[0.0849, 0.2241]	0.0262	0.0703	[0.0271, 0.0703]	0.0407
synth_dev_unseen_dental_practice@lt	113	0.1047	0.2181	[0.0968, 0.2181]	0.0453	0.0807	[0.0394, 0.0807]	0.0772
synth_dev_unseen_dental_practice@pl	117	0.1245	0.2415	[0.1047, 0.2415]	0.0363	0.0711	[0.0356, 0.0711]	0.0384
synth_dev_unseen_dental_practice@sk	116	0.0763	0.2119	[0.0737, 0.2119]	0.0256	0.0671	[0.0256, 0.0671]	0.0342
synth_dev_unseen_ebook_platform	128	0.0823	0.1496	[0.0690, 0.1496]	0.0229	0.0509	[0.0206, 0.0509]	0.0297
synth_dev_unseen_ebook_platform@de	102	0.1056	0.1836	[0.0795, 0.1836]	0.0420	0.0701	[0.0312, 0.0701]	0.0221
synth_dev_unseen_ebook_platform@lt	104	0.0828	0.1655	[0.0748, 0.1655]	0.0209	0.0519	[0.0224, 0.0519]	0.0370
synth_dev_unseen_ebook_platform@pl	100	0.0770	0.1953	[0.0747, 0.1953]	0.0208	0.0576	[0.0222, 0.0576]	0.0236
synth_dev_unseen_ebook_platform@sk	100	0.0890	0.2042	[0.0731, 0.2042]	0.0236	0.0659	[0.0232, 0.0659]	0.0448
synth_dev_unseen_home_appliance_repair	81	0.1219	n/a	[0.0952, 0.1719]	0.0449	n/a	[0.0346, 0.0695]	0.0128
synth_dev_unseen_home_appliance_repair@de	67	0.0828	n/a	[0.0723, 0.2184]	0.0298	n/a	[0.0296, 0.0673]	0.0195
synth_dev_unseen_home_appliance_repair@lt	66	0.1375	n/a	[0.1167, 0.2479]	0.0497	n/a	[0.0445, 0.0791]	0.0316
synth_dev_unseen_home_appliance_repair@pl	63	0.1461	n/a	[0.1105, 0.2297]	0.0485	n/a	[0.0400, 0.0783]	0.0310
synth_dev_unseen_home_appliance_repair@sk	67	0.1309	n/a	[0.1129, 0.2152]	0.0444	n/a	[0.0432, 0.0706]	0.0539
synth_dev_unseen_online_auction_site	117	0.0785	0.1462	[0.0659, 0.1462]	0.0253	0.0506	[0.0221, 0.0506]	0.0137
synth_dev_unseen_online_auction_site@de	83	0.1199	0.1874	[0.0899, 0.1874]	0.0386	0.0740	[0.0288, 0.0740]	0.0248
synth_dev_unseen_online_auction_site@lt	93	0.1395	0.2058	[0.0999, 0.2058]	0.0468	0.0758	[0.0388, 0.0758]	0.0438
synth_dev_unseen_online_auction_site@pl	88	0.0940	0.1975	[0.0794, 0.1975]	0.0320	0.0643	[0.0277, 0.0643]	0.0165
synth_dev_unseen_online_auction_site@sk	90	0.1389	0.2176	[0.1271, 0.2176]	0.0462	0.0773	[0.0407, 0.0773]	0.0178
synth_dev_unseen_student_services_office	177	0.0706	0.1365	[0.0565, 0.1365]	0.0222	0.0436	[0.0183, 0.0436]	0.0309
synth_dev_unseen_tattoo_studio	114	0.1400	0.2880	[0.1089, 0.2880]	0.0482	0.1013	[0.0363, 0.1013]	0.0484
synth_dev_unseen_tattoo_studio@de	86	0.1424	0.2772	[0.1141, 0.2772]	0.0586	0.0949	[0.0521, 0.0949]	0.1077
synth_dev_unseen_tattoo_studio@lt	87	0.1478	0.2921	[0.1153, 0.2921]	0.0602	0.1026	[0.0491, 0.1026]	0.1099
synth_dev_unseen_tattoo_studio@pl	93	0.1054	0.2874	[0.1053, 0.2874]	0.0496	0.0841	[0.0506, 0.0841]	0.0975
synth_dev_unseen_tattoo_studio@sk	90	0.1482	0.3002	[0.1165, 0.3002]	0.0560	0.1027	[0.0502, 0.1027]	0.0860
synth_dev_unseen_vpn_service	65	0.1411	n/a	[0.0957, 0.2312]	0.0450	n/a	[0.0308, 0.0767]	0.0170
synth_dev_unseen_vpn_service@de	52	0.1751	n/a	[0.1097, 0.2801]	0.0503	n/a	[0.0361, 0.0936]	0.0211
synth_dev_unseen_vpn_service@lt	48	0.1782	n/a	[0.1549, 0.2862]	0.0584	n/a	[0.0502, 0.1034]	0.0285
synth_dev_unseen_vpn_service@pl	49	0.2091	n/a	[0.1473, 0.2836]	0.0629	n/a	[0.0468, 0.0978]	0.0267
synth_dev_unseen_vpn_service@sk	50	0.2121	n/a	[0.1359, 0.3145]	0.0645	n/a	[0.0442, 0.1128]	0.0274
synth_dev_unseen_waste_collection_service	168	0.1023	0.1952	[0.0895, 0.1952]	0.0314	0.0633	[0.0291, 0.0633]	0.0703
synth_dev_unseen_yoga_studio	132	0.0746	0.2107	[0.0710, 0.2107]	0.0297	0.0697	[0.0277, 0.0697]	0.0378
synth_dev_unseen_yoga_studio@de	107	0.1202	0.2389	[0.0984, 0.2389]	0.0330	0.0730	[0.030005, 0.0730]	0.0486
synth_dev_unseen_yoga_studio@lt	104	0.0960	0.2619	[0.0895, 0.2619]	0.0379	0.0875	[0.0339, 0.0875]	0.0533
synth_dev_unseen_yoga_studio@pl	101	0.1547	0.2941	[0.1170, 0.2941]	0.0562	0.0929	[0.0480, 0.0929]	0.1151
synth_dev_unseen_yoga_studio@sk	101	0.1201	0.2583	[0.0943, 0.2583]	0.0434	0.0841	[0.0350, 0.0841]	0.0809

information.by_pool	scores.rem_ece_p975	scores.rem_ece_bands.6 to 20	scores.rem_ece_bands.21 to 40	scores.rem_ece_bands.41 or more	scores.remainder_info.recall_at_5	scores.remainder_info.over_5.rows	scores.remainder_info.over_5.not_read_share	scores.remainder_info.over_5.mean_remainder
abcd_dev	0.1701	n/a	n/a	0.0883	0.7897	504	0.2103	0.1290
abcd_dev@de	0.1995	n/a	n/a	0.1000	0.7692	260	0.2308	0.1425
abcd_dev@lt	0.2004	n/a	n/a	0.1048	0.7585	265	0.2415	0.1608
abcd_dev@pl	0.1884	n/a	n/a	0.0956	0.7366	243	0.2634	0.1677
abcd_dev@sk	0.1878	n/a	n/a	0.0990	0.7391	253	0.2609	0.1653
bitext_dev	0.0232	n/a	0.0136	n/a	1.0000	508	0.0000	0.0136
clinc_dev@de	0.0693	n/a	n/a	0.0360	0.9655	464	0.0345	0.0703
clinc_dev@lt	0.0899	n/a	n/a	0.0525	0.9465	486	0.0535	0.1036
clinc_dev@pl	0.0783	n/a	n/a	0.0487	0.9457	479	0.0543	0.0949
clinc_dev@sk	0.0858	n/a	n/a	0.0558	0.9544	482	0.0456	0.0995
dstc11_banking_guard	0.0067	0.0047	n/a	n/a	1.0000	167	0.0000	0.0047
dstc11_banking_guard@de	0.0098	0.0069	n/a	n/a	1.0000	65	0.0000	0.0069
dstc11_banking_guard@lt	0.0135	0.0094	n/a	n/a	1.0000	62	0.0000	0.0094
dstc11_banking_guard@pl	0.0182	0.0103	n/a	n/a	1.0000	51	0.0000	0.0103
dstc11_banking_guard@sk	0.0136	0.0086	n/a	n/a	1.0000	51	0.0000	0.0086
dstc11_finance_guard	0.0285	n/a	0.0119	n/a	0.98996	498	0.01004	0.0219
dstc11_finance_guard@de	0.0644	n/a	0.0329	n/a	0.9758	165	0.0242	0.0394
dstc11_finance_guard@lt	0.0587	n/a	0.0190	n/a	0.9639	166	0.0361	0.0436
dstc11_finance_guard@pl	0.0653	n/a	0.0261	n/a	0.9504	141	0.0496	0.0411
dstc11_finance_guard@sk	0.0677	n/a	0.0292	n/a	0.9669	151	0.0331	0.0438
dstc11_insurance	0.0160	n/a	0.0096	n/a	0.9973	377	0.0027	0.0099
hint3_dev	n/a	0.1078	n/a	n/a	0.8806	67	0.1194	0.0151
ind_dev_unseen_electricity_grid_operator	0.0622	n/a	0.0410	n/a	0.9728	147	0.0272	0.0501
ind_dev_unseen_electricity_grid_operator@lt	0.0845	n/a	0.0391	n/a	0.9406	101	0.0594	0.0865
ind_dev_unseen_electricity_grid_operator@pl	0.1010	n/a	0.0341	n/a	0.8947	95	0.1053	0.0910
ind_dev_unseen_property_management_company	0.0641	n/a	n/a	0.0196	0.9419	258	0.0581	0.0697
ind_dev_unseen_property_management_company@lt	0.1301	n/a	n/a	0.0622	0.8850	200	0.1150	0.1131
ind_dev_unseen_property_management_company@pl	0.0986	n/a	n/a	0.0316	0.8859	184	0.1141	0.1174
ind_dev_unseen_tax_and_social_security_office	0.0698	n/a	n/a	0.0437	0.9634	246	0.0366	0.0701
ind_dev_unseen_tax_and_social_security_office@lt	0.1184	n/a	n/a	0.0914	0.9746	197	0.0254	0.1108
ind_dev_unseen_tax_and_social_security_office@pl	0.0988	n/a	n/a	0.0599	0.9531	192	0.0469	0.1068
ind_dev_unseen_telecom_operator	0.0540	n/a	0.0272	n/a	0.9739	153	0.0261	0.0431
ind_dev_unseen_telecom_operator@lt	0.0841	n/a	0.0374	n/a	0.9407	118	0.0593	0.0739
ind_dev_unseen_telecom_operator@pl	0.1144	n/a	0.0642	n/a	0.9304	115	0.0696	0.0755
sgd_unseen	n/a	0.0010	n/a	n/a	1.0000	102	0.0000	0.0010
synth_dev_seen_domain	0.0693	n/a	n/a	0.0453	0.9629	377	0.0371	0.0762
synth_dev_seen_domain@de	0.0688	n/a	n/a	0.0341	0.9465	299	0.0535	0.0869
synth_dev_seen_domain@lt	0.0767	n/a	n/a	0.0424	0.9281	306	0.0719	0.1134
synth_dev_seen_domain@pl	0.0915	n/a	n/a	0.0597	0.9381	291	0.0619	0.1089
synth_dev_seen_domain@sk	0.1075	n/a	n/a	0.0739	0.9463	298	0.0537	0.1122
synth_dev_unseen_bookstore	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_bookstore@de	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_bookstore@lt	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_bookstore@pl	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_bookstore@sk	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_camera_store	n/a	0.0088	n/a	n/a	1.0000	45	0.0000	0.0088
synth_dev_unseen_car_dealership_service_department	n/a	0.0152	n/a	n/a	1.0000	57	0.0000	0.0152
synth_dev_unseen_car_wash_subscription	0.1198	0.0305	n/a	n/a	0.9452	73	0.0548	0.0286
synth_dev_unseen_dental_practice	0.0966	n/a	0.0436	n/a	0.9527	148	0.0473	0.0608
synth_dev_unseen_dental_practice@de	0.0766	n/a	0.0407	n/a	0.9565	115	0.0435	0.0727
synth_dev_unseen_dental_practice@lt	0.1495	n/a	0.0772	n/a	0.9204	113	0.0796	0.0827
synth_dev_unseen_dental_practice@pl	0.0978	n/a	0.0384	n/a	0.8974	117	0.1026	0.0897
synth_dev_unseen_dental_practice@sk	0.1114	n/a	0.0342	n/a	0.9138	116	0.0862	0.0834
synth_dev_unseen_ebook_platform	0.0442	0.0297	n/a	n/a	1.0000	128	0.0000	0.0297
synth_dev_unseen_ebook_platform@de	0.0437	0.0221	n/a	n/a	0.9902	102	0.0098	0.0319
synth_dev_unseen_ebook_platform@lt	0.0594	0.0370	n/a	n/a	0.9904	104	0.0096	0.0466
synth_dev_unseen_ebook_platform@pl	0.1119	0.0236	n/a	n/a	0.9300	100	0.0700	0.0485
synth_dev_unseen_ebook_platform@sk	0.0774	0.0448	n/a	n/a	0.9700	100	0.0300	0.0431
synth_dev_unseen_home_appliance_repair	n/a	0.0128	n/a	n/a	0.9877	81	0.0123	0.0252
synth_dev_unseen_home_appliance_repair@de	n/a	0.0195	n/a	n/a	0.9552	67	0.0448	0.0301
synth_dev_unseen_home_appliance_repair@lt	n/a	0.0316	n/a	n/a	0.9242	66	0.0758	0.0441
synth_dev_unseen_home_appliance_repair@pl	n/a	0.0310	n/a	n/a	0.9524	63	0.0476	0.0435
synth_dev_unseen_home_appliance_repair@sk	n/a	0.0539	n/a	n/a	0.9403	67	0.0597	0.0428
synth_dev_unseen_online_auction_site	0.0434	0.0137	n/a	n/a	0.9829	117	0.0171	0.0270
synth_dev_unseen_online_auction_site@de	0.0480	0.0248	n/a	n/a	0.9880	83	0.0120	0.0273
synth_dev_unseen_online_auction_site@lt	0.0584	0.0438	n/a	n/a	1.0000	93	0.0000	0.0438
synth_dev_unseen_online_auction_site@pl	0.0828	0.0165	n/a	n/a	0.9545	88	0.0455	0.0440
synth_dev_unseen_online_auction_site@sk	0.0680	0.0178	n/a	n/a	0.9556	90	0.0444	0.0484
synth_dev_unseen_student_services_office	0.0507	n/a	0.0309	n/a	0.9774	177	0.0226	0.0374
synth_dev_unseen_tattoo_studio	0.1325	0.0484	n/a	n/a	0.9123	114	0.0877	0.0748
synth_dev_unseen_tattoo_studio@de	0.3105	0.1077	n/a	n/a	0.8256	86	0.1744	0.0784
synth_dev_unseen_tattoo_studio@lt	0.2907	0.1099	n/a	n/a	0.7931	87	0.2069	0.0970
synth_dev_unseen_tattoo_studio@pl	0.2841	0.0975	n/a	n/a	0.8280	93	0.1720	0.0921
synth_dev_unseen_tattoo_studio@sk	0.2522	0.0860	n/a	n/a	0.8222	90	0.1778	0.0918
synth_dev_unseen_vpn_service	n/a	0.0170	n/a	n/a	1.0000	65	0.0000	0.0170
synth_dev_unseen_vpn_service@de	n/a	0.0211	n/a	n/a	1.0000	52	0.0000	0.0211
synth_dev_unseen_vpn_service@lt	n/a	0.0285	n/a	n/a	1.0000	48	0.0000	0.0285
synth_dev_unseen_vpn_service@pl	n/a	0.0267	n/a	n/a	1.0000	49	0.0000	0.0267
synth_dev_unseen_vpn_service@sk	n/a	0.0274	n/a	n/a	0.9800	50	0.0200	0.0283
synth_dev_unseen_waste_collection_service	0.1455	n/a	0.0703	n/a	0.8631	168	0.1369	0.1095
synth_dev_unseen_yoga_studio	0.1198	n/a	0.0378	n/a	0.8864	132	0.1136	0.0758
synth_dev_unseen_yoga_studio@de	0.1430	n/a	0.0486	n/a	0.8972	107	0.1028	0.0720
synth_dev_unseen_yoga_studio@lt	0.1354	n/a	0.0533	n/a	0.8558	104	0.1442	0.0961
synth_dev_unseen_yoga_studio@pl	0.2544	n/a	0.1151	n/a	0.7921	101	0.2079	0.0929
synth_dev_unseen_yoga_studio@sk	0.1814	n/a	0.0809	n/a	0.8218	101	0.1782	0.0981

information.by_pool	scores.remainder_info.over_5.remainder_ece	scores.remainder_info.by_size.6 to 20.rows	scores.remainder_info.by_size.6 to 20.not_read_share	scores.remainder_info.by_size.6 to 20.mean_remainder	scores.remainder_info.by_size.6 to 20.remainder_ece	scores.remainder_info.by_size.21 to 40.rows	scores.remainder_info.by_size.21 to 40.not_read_share	scores.remainder_info.by_size.21 to 40.mean_remainder
abcd_dev	0.0883	0	n/a	n/a	n/a	0	n/a	n/a
abcd_dev@de	0.1000	0	n/a	n/a	n/a	0	n/a	n/a
abcd_dev@lt	0.1048	0	n/a	n/a	n/a	0	n/a	n/a
abcd_dev@pl	0.0956	0	n/a	n/a	n/a	0	n/a	n/a
abcd_dev@sk	0.0990	0	n/a	n/a	n/a	0	n/a	n/a
bitext_dev	0.0136	0	n/a	n/a	n/a	508	0.0000	0.0136
clinc_dev@de	0.0360	0	n/a	n/a	n/a	0	n/a	n/a
clinc_dev@lt	0.0525	0	n/a	n/a	n/a	0	n/a	n/a
clinc_dev@pl	0.0487	0	n/a	n/a	n/a	0	n/a	n/a
clinc_dev@sk	0.0558	0	n/a	n/a	n/a	0	n/a	n/a
dstc11_banking_guard	0.0047	167	0.0000	0.0047	0.0047	0	n/a	n/a
dstc11_banking_guard@de	0.0069	65	0.0000	0.0069	0.0069	0	n/a	n/a
dstc11_banking_guard@lt	0.0094	62	0.0000	0.0094	0.0094	0	n/a	n/a
dstc11_banking_guard@pl	0.0103	51	0.0000	0.0103	0.0103	0	n/a	n/a
dstc11_banking_guard@sk	0.0086	51	0.0000	0.0086	0.0086	0	n/a	n/a
dstc11_finance_guard	0.0119	0	n/a	n/a	n/a	498	0.01004	0.0219
dstc11_finance_guard@de	0.0329	0	n/a	n/a	n/a	165	0.0242	0.0394
dstc11_finance_guard@lt	0.0190	0	n/a	n/a	n/a	166	0.0361	0.0436
dstc11_finance_guard@pl	0.0261	0	n/a	n/a	n/a	141	0.0496	0.0411
dstc11_finance_guard@sk	0.0292	0	n/a	n/a	n/a	151	0.0331	0.0438
dstc11_insurance	0.0096	0	n/a	n/a	n/a	377	0.0027	0.0099
hint3_dev	0.1078	67	0.1194	0.0151	0.1078	0	n/a	n/a
ind_dev_unseen_electricity_grid_operator	0.0410	0	n/a	n/a	n/a	147	0.0272	0.0501
ind_dev_unseen_electricity_grid_operator@lt	0.0391	0	n/a	n/a	n/a	101	0.0594	0.0865
ind_dev_unseen_electricity_grid_operator@pl	0.0341	0	n/a	n/a	n/a	95	0.1053	0.0910
ind_dev_unseen_property_management_company	0.0196	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_property_management_company@lt	0.0622	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_property_management_company@pl	0.0316	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_tax_and_social_security_office	0.0437	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_tax_and_social_security_office@lt	0.0914	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_tax_and_social_security_office@pl	0.0599	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_telecom_operator	0.0272	0	n/a	n/a	n/a	153	0.0261	0.0431
ind_dev_unseen_telecom_operator@lt	0.0374	0	n/a	n/a	n/a	118	0.0593	0.0739
ind_dev_unseen_telecom_operator@pl	0.0642	0	n/a	n/a	n/a	115	0.0696	0.0755
sgd_unseen	0.0010	102	0.0000	0.0010	0.0010	0	n/a	n/a
synth_dev_seen_domain	0.0453	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_seen_domain@de	0.0341	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_seen_domain@lt	0.0424	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_seen_domain@pl	0.0597	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_seen_domain@sk	0.0739	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore@de	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore@lt	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore@pl	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore@sk	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_camera_store	0.0088	45	0.0000	0.0088	0.0088	0	n/a	n/a
synth_dev_unseen_car_dealership_service_department	0.0152	57	0.0000	0.0152	0.0152	0	n/a	n/a
synth_dev_unseen_car_wash_subscription	0.0305	73	0.0548	0.0286	0.0305	0	n/a	n/a
synth_dev_unseen_dental_practice	0.0436	0	n/a	n/a	n/a	148	0.0473	0.0608
synth_dev_unseen_dental_practice@de	0.0407	0	n/a	n/a	n/a	115	0.0435	0.0727
synth_dev_unseen_dental_practice@lt	0.0772	0	n/a	n/a	n/a	113	0.0796	0.0827
synth_dev_unseen_dental_practice@pl	0.0384	0	n/a	n/a	n/a	117	0.1026	0.0897
synth_dev_unseen_dental_practice@sk	0.0342	0	n/a	n/a	n/a	116	0.0862	0.0834
synth_dev_unseen_ebook_platform	0.0297	128	0.0000	0.0297	0.0297	0	n/a	n/a
synth_dev_unseen_ebook_platform@de	0.0221	102	0.0098	0.0319	0.0221	0	n/a	n/a
synth_dev_unseen_ebook_platform@lt	0.0370	104	0.0096	0.0466	0.0370	0	n/a	n/a
synth_dev_unseen_ebook_platform@pl	0.0236	100	0.0700	0.0485	0.0236	0	n/a	n/a
synth_dev_unseen_ebook_platform@sk	0.0448	100	0.0300	0.0431	0.0448	0	n/a	n/a
synth_dev_unseen_home_appliance_repair	0.0128	81	0.0123	0.0252	0.0128	0	n/a	n/a
synth_dev_unseen_home_appliance_repair@de	0.0195	67	0.0448	0.0301	0.0195	0	n/a	n/a
synth_dev_unseen_home_appliance_repair@lt	0.0316	66	0.0758	0.0441	0.0316	0	n/a	n/a
synth_dev_unseen_home_appliance_repair@pl	0.0310	63	0.0476	0.0435	0.0310	0	n/a	n/a
synth_dev_unseen_home_appliance_repair@sk	0.0539	67	0.0597	0.0428	0.0539	0	n/a	n/a
synth_dev_unseen_online_auction_site	0.0137	117	0.0171	0.0270	0.0137	0	n/a	n/a
synth_dev_unseen_online_auction_site@de	0.0248	83	0.0120	0.0273	0.0248	0	n/a	n/a
synth_dev_unseen_online_auction_site@lt	0.0438	93	0.0000	0.0438	0.0438	0	n/a	n/a
synth_dev_unseen_online_auction_site@pl	0.0165	88	0.0455	0.0440	0.0165	0	n/a	n/a
synth_dev_unseen_online_auction_site@sk	0.0178	90	0.0444	0.0484	0.0178	0	n/a	n/a
synth_dev_unseen_student_services_office	0.0309	0	n/a	n/a	n/a	177	0.0226	0.0374
synth_dev_unseen_tattoo_studio	0.0484	114	0.0877	0.0748	0.0484	0	n/a	n/a
synth_dev_unseen_tattoo_studio@de	0.1077	86	0.1744	0.0784	0.1077	0	n/a	n/a
synth_dev_unseen_tattoo_studio@lt	0.1099	87	0.2069	0.0970	0.1099	0	n/a	n/a
synth_dev_unseen_tattoo_studio@pl	0.0975	93	0.1720	0.0921	0.0975	0	n/a	n/a
synth_dev_unseen_tattoo_studio@sk	0.0860	90	0.1778	0.0918	0.0860	0	n/a	n/a
synth_dev_unseen_vpn_service	0.0170	65	0.0000	0.0170	0.0170	0	n/a	n/a
synth_dev_unseen_vpn_service@de	0.0211	52	0.0000	0.0211	0.0211	0	n/a	n/a
synth_dev_unseen_vpn_service@lt	0.0285	48	0.0000	0.0285	0.0285	0	n/a	n/a
synth_dev_unseen_vpn_service@pl	0.0267	49	0.0000	0.0267	0.0267	0	n/a	n/a
synth_dev_unseen_vpn_service@sk	0.0274	50	0.0200	0.0283	0.0274	0	n/a	n/a
synth_dev_unseen_waste_collection_service	0.0703	0	n/a	n/a	n/a	168	0.1369	0.1095
synth_dev_unseen_yoga_studio	0.0378	0	n/a	n/a	n/a	132	0.1136	0.0758
synth_dev_unseen_yoga_studio@de	0.0486	0	n/a	n/a	n/a	107	0.1028	0.0720
synth_dev_unseen_yoga_studio@lt	0.0533	0	n/a	n/a	n/a	104	0.1442	0.0961
synth_dev_unseen_yoga_studio@pl	0.1151	0	n/a	n/a	n/a	101	0.2079	0.0929
synth_dev_unseen_yoga_studio@sk	0.0809	0	n/a	n/a	n/a	101	0.1782	0.0981

information.by_pool	scores.remainder_info.by_size.21 to 40.remainder_ece	scores.remainder_info.by_size.41 or more.rows	scores.remainder_info.by_size.41 or more.not_read_share	scores.remainder_info.by_size.41 or more.mean_remainder	scores.remainder_info.by_size.41 or more.remainder_ece	scores.rem_ece_band_intervals.6 to 20	scores.rem_ece_band_intervals.21 to 40	scores.rem_ece_band_intervals.41 or more
abcd_dev	n/a	504	0.2103	0.1290	0.0883	n/a	n/a	[0.0411, 0.1701]
abcd_dev@de	n/a	260	0.2308	0.1425	0.1000	n/a	n/a	[0.0457, 0.1995]
abcd_dev@lt	n/a	265	0.2415	0.1608	0.1048	n/a	n/a	[0.0648, 0.2004]
abcd_dev@pl	n/a	243	0.2634	0.1677	0.0956	n/a	n/a	[0.0594, 0.1884]
abcd_dev@sk	n/a	253	0.2609	0.1653	0.0990	n/a	n/a	[0.0515, 0.1878]
bitext_dev	0.0136	0	n/a	n/a	n/a	n/a	[0.0051, 0.0232]	n/a
clinc_dev@de	n/a	464	0.0345	0.0703	0.0360	n/a	n/a	[0.0187, 0.0693]
clinc_dev@lt	n/a	486	0.0535	0.1036	0.0525	n/a	n/a	[0.0345, 0.0899]
clinc_dev@pl	n/a	479	0.0543	0.0949	0.0487	n/a	n/a	[0.0363, 0.0783]
clinc_dev@sk	n/a	482	0.0456	0.0995	0.0558	n/a	n/a	[0.0384, 0.0858]
dstc11_banking_guard	n/a	0	n/a	n/a	n/a	[0.0031, 0.0067]	n/a	n/a
dstc11_banking_guard@de	n/a	0	n/a	n/a	n/a	[0.0038, 0.0098]	n/a	n/a
dstc11_banking_guard@lt	n/a	0	n/a	n/a	n/a	[0.0060, 0.0135]	n/a	n/a
dstc11_banking_guard@pl	n/a	0	n/a	n/a	n/a	[0.0044, 0.0182]	n/a	n/a
dstc11_banking_guard@sk	n/a	0	n/a	n/a	n/a	[0.0048, 0.0136]	n/a	n/a
dstc11_finance_guard	0.0119	0	n/a	n/a	n/a	n/a	[0.0064, 0.0285]	n/a
dstc11_finance_guard@de	0.0329	0	n/a	n/a	n/a	n/a	[0.0156, 0.0644]	n/a
dstc11_finance_guard@lt	0.0190	0	n/a	n/a	n/a	n/a	[0.0117, 0.0587]	n/a
dstc11_finance_guard@pl	0.0261	0	n/a	n/a	n/a	n/a	[0.0159, 0.0653]	n/a
dstc11_finance_guard@sk	0.0292	0	n/a	n/a	n/a	n/a	[0.0175, 0.0677]	n/a
dstc11_insurance	0.0096	0	n/a	n/a	n/a	n/a	[0.0063, 0.0160]	n/a
hint3_dev	n/a	0	n/a	n/a	n/a	[0.0038, 0.2910]	n/a	n/a
ind_dev_unseen_electricity_grid_operator	0.0410	0	n/a	n/a	n/a	n/a	[0.0255, 0.0622]	n/a
ind_dev_unseen_electricity_grid_operator@lt	0.0391	0	n/a	n/a	n/a	n/a	[0.0325, 0.0845]	n/a
ind_dev_unseen_electricity_grid_operator@pl	0.0341	0	n/a	n/a	n/a	n/a	[0.0209, 0.1010]	n/a
ind_dev_unseen_property_management_company	n/a	258	0.0581	0.0697	0.0196	n/a	n/a	[0.0188, 0.0641]
ind_dev_unseen_property_management_company@lt	n/a	200	0.1150	0.1131	0.0622	n/a	n/a	[0.0412, 0.1301]
ind_dev_unseen_property_management_company@pl	n/a	184	0.1141	0.1174	0.0316	n/a	n/a	[0.0283, 0.0986]
ind_dev_unseen_tax_and_social_security_office	n/a	246	0.0366	0.0701	0.0437	n/a	n/a	[0.0336, 0.0698]
ind_dev_unseen_tax_and_social_security_office@lt	n/a	197	0.0254	0.1108	0.0914	n/a	n/a	[0.0737, 0.1184]
ind_dev_unseen_tax_and_social_security_office@pl	n/a	192	0.0469	0.1068	0.0599	n/a	n/a	[0.0410, 0.0988]
ind_dev_unseen_telecom_operator	0.0272	0	n/a	n/a	n/a	n/a	[0.0130, 0.0540]	n/a
ind_dev_unseen_telecom_operator@lt	0.0374	0	n/a	n/a	n/a	n/a	[0.0197, 0.0841]	n/a
ind_dev_unseen_telecom_operator@pl	0.0642	0	n/a	n/a	n/a	n/a	[0.0337, 0.1144]	n/a
sgd_unseen	n/a	0	n/a	n/a	n/a	[0.0002, 0.0027]	n/a	n/a
synth_dev_seen_domain	n/a	377	0.0371	0.0762	0.0453	n/a	n/a	[0.0335, 0.0693]
synth_dev_seen_domain@de	n/a	299	0.0535	0.0869	0.0341	n/a	n/a	[0.0254, 0.0688]
synth_dev_seen_domain@lt	n/a	306	0.0719	0.1134	0.0424	n/a	n/a	[0.0326, 0.0767]
synth_dev_seen_domain@pl	n/a	291	0.0619	0.1089	0.0597	n/a	n/a	[0.0443, 0.0915]
synth_dev_seen_domain@sk	n/a	298	0.0537	0.1122	0.0739	n/a	n/a	[0.0519, 0.1075]
synth_dev_unseen_bookstore	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_bookstore@de	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_bookstore@lt	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_bookstore@pl	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_bookstore@sk	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_camera_store	n/a	0	n/a	n/a	n/a	[0.0028, 0.0140]	n/a	n/a
synth_dev_unseen_car_dealership_service_department	n/a	0	n/a	n/a	n/a	[0.0088, 0.0248]	n/a	n/a
synth_dev_unseen_car_wash_subscription	n/a	0	n/a	n/a	n/a	[0.0168, 0.1198]	n/a	n/a
synth_dev_unseen_dental_practice	0.0436	0	n/a	n/a	n/a	n/a	[0.0289, 0.0966]	n/a
synth_dev_unseen_dental_practice@de	0.0407	0	n/a	n/a	n/a	n/a	[0.0258, 0.0766]	n/a
synth_dev_unseen_dental_practice@lt	0.0772	0	n/a	n/a	n/a	n/a	[0.0449, 0.1495]	n/a
synth_dev_unseen_dental_practice@pl	0.0384	0	n/a	n/a	n/a	n/a	[0.0212, 0.0978]	n/a
synth_dev_unseen_dental_practice@sk	0.0342	0	n/a	n/a	n/a	n/a	[0.0283, 0.1114]	n/a
synth_dev_unseen_ebook_platform	n/a	0	n/a	n/a	n/a	[0.0175, 0.0442]	n/a	n/a
synth_dev_unseen_ebook_platform@de	n/a	0	n/a	n/a	n/a	[0.0123, 0.0437]	n/a	n/a
synth_dev_unseen_ebook_platform@lt	n/a	0	n/a	n/a	n/a	[0.0228, 0.0594]	n/a	n/a
synth_dev_unseen_ebook_platform@pl	n/a	0	n/a	n/a	n/a	[0.0111, 0.1119]	n/a	n/a
synth_dev_unseen_ebook_platform@sk	n/a	0	n/a	n/a	n/a	[0.0265, 0.0774]	n/a	n/a
synth_dev_unseen_home_appliance_repair	n/a	0	n/a	n/a	n/a	[0.0088, 0.0405]	n/a	n/a
synth_dev_unseen_home_appliance_repair@de	n/a	0	n/a	n/a	n/a	[0.0094, 0.0944]	n/a	n/a
synth_dev_unseen_home_appliance_repair@lt	n/a	0	n/a	n/a	n/a	[0.0129, 0.1078]	n/a	n/a
synth_dev_unseen_home_appliance_repair@pl	n/a	0	n/a	n/a	n/a	[0.0174, 0.0962]	n/a	n/a
synth_dev_unseen_home_appliance_repair@sk	n/a	0	n/a	n/a	n/a	[0.0226, 0.1457]	n/a	n/a
synth_dev_unseen_online_auction_site	n/a	0	n/a	n/a	n/a	[0.0132, 0.0434]	n/a	n/a
synth_dev_unseen_online_auction_site@de	n/a	0	n/a	n/a	n/a	[0.0169, 0.0480]	n/a	n/a
synth_dev_unseen_online_auction_site@lt	n/a	0	n/a	n/a	n/a	[0.0311, 0.0584]	n/a	n/a
synth_dev_unseen_online_auction_site@pl	n/a	0	n/a	n/a	n/a	[0.0098, 0.0828]	n/a	n/a
synth_dev_unseen_online_auction_site@sk	n/a	0	n/a	n/a	n/a	[0.0120, 0.0680]	n/a	n/a
synth_dev_unseen_student_services_office	0.0309	0	n/a	n/a	n/a	n/a	[0.0216, 0.0507]	n/a
synth_dev_unseen_tattoo_studio	n/a	0	n/a	n/a	n/a	[0.0178, 0.1325]	n/a	n/a
synth_dev_unseen_tattoo_studio@de	n/a	0	n/a	n/a	n/a	[0.0412, 0.3105]	n/a	n/a
synth_dev_unseen_tattoo_studio@lt	n/a	0	n/a	n/a	n/a	[0.0294, 0.2907]	n/a	n/a
synth_dev_unseen_tattoo_studio@pl	n/a	0	n/a	n/a	n/a	[0.0242, 0.2841]	n/a	n/a
synth_dev_unseen_tattoo_studio@sk	n/a	0	n/a	n/a	n/a	[0.0287, 0.2522]	n/a	n/a
synth_dev_unseen_vpn_service	n/a	0	n/a	n/a	n/a	[0.0077, 0.0262]	n/a	n/a
synth_dev_unseen_vpn_service@de	n/a	0	n/a	n/a	n/a	[0.0070, 0.0400]	n/a	n/a
synth_dev_unseen_vpn_service@lt	n/a	0	n/a	n/a	n/a	[0.0118, 0.0479]	n/a	n/a
synth_dev_unseen_vpn_service@pl	n/a	0	n/a	n/a	n/a	[0.0118, 0.0432]	n/a	n/a
synth_dev_unseen_vpn_service@sk	n/a	0	n/a	n/a	n/a	[0.0125, 0.0814]	n/a	n/a
synth_dev_unseen_waste_collection_service	0.0703	0	n/a	n/a	n/a	n/a	[0.0367, 0.1455]	n/a
synth_dev_unseen_yoga_studio	0.0378	0	n/a	n/a	n/a	n/a	[0.0194, 0.1198]	n/a
synth_dev_unseen_yoga_studio@de	0.0486	0	n/a	n/a	n/a	n/a	[0.0198, 0.1430]	n/a
synth_dev_unseen_yoga_studio@lt	0.0533	0	n/a	n/a	n/a	n/a	[0.0270, 0.1354]	n/a
synth_dev_unseen_yoga_studio@pl	0.1151	0	n/a	n/a	n/a	n/a	[0.0339, 0.2544]	n/a
synth_dev_unseen_yoga_studio@sk	0.0809	0	n/a	n/a	n/a	n/a	[0.0362, 0.1814]	n/a

information.by_pool	scores.rem_ece_interval	scores.recall_at_5	scores.recall_at_5_interval	scores.accuracy	scores.accuracy_interval	scores.mean_top	scores.mean_top_interval	scores.mean_top_minus_accuracy
abcd_dev	[0.0411, 0.1701]	0.7897	[0.7060, 0.8648]	0.5694	[0.4751, 0.6597]	0.6658	[0.6192, 0.7108]	0.0963
abcd_dev@de	[0.0457, 0.1995]	0.7692	[0.6706, 0.8559]	0.5462	[0.4395, 0.6417]	0.6358	[0.5899, 0.6801]	0.0897
abcd_dev@lt	[0.0648, 0.2004]	0.7585	[0.6618, 0.8399]	0.5057	[0.3991, 0.6059]	0.5944	[0.5454, 0.6413]	0.0888
abcd_dev@pl	[0.0594, 0.1884]	0.7366	[0.6509, 0.8169]	0.5062	[0.3966, 0.6068]	0.5915	[0.5444, 0.6412]	0.0854
abcd_dev@sk	[0.0515, 0.1878]	0.7391	[0.6556, 0.8152]	0.5178	[0.4206, 0.6108]	0.5936	[0.5378, 0.6498]	0.0758
bitext_dev	[0.0051, 0.0232]	1.0000	[1.0000, 1.0000]	0.9232	[0.8504, 0.9861]	0.8846	[0.8224, 0.9432]	-0.0386
clinc_dev@de	[0.0187, 0.0693]	0.9655	[0.9264, 0.9927]	0.8233	[0.7404, 0.8978]	0.7917	[0.7403, 0.8425]	-0.0316
clinc_dev@lt	[0.0345, 0.0899]	0.9465	[0.9052, 0.9799]	0.7449	[0.6602, 0.8257]	0.7226	[0.6595, 0.7802]	-0.0223
clinc_dev@pl	[0.0363, 0.0783]	0.9457	[0.8957, 0.9819]	0.7662	[0.6834, 0.8411]	0.7373	[0.6829, 0.7895]	-0.0289
clinc_dev@sk	[0.0384, 0.0858]	0.9544	[0.9170, 0.9833]	0.7510	[0.6642, 0.8309]	0.7345	[0.6759, 0.7911]	-0.0166
dstc11_banking_guard	[0.0031, 0.0067]	1.0000	[1.0000, 1.0000]	0.9880	[0.9695, 1.0000]	0.9483	[0.9300, 0.9644]	-0.0397
dstc11_banking_guard@de	[0.0038, 0.0098]	1.0000	[1.0000, 1.0000]	0.9692	[0.9242, 1.0000]	0.9001	[0.8276, 0.9633]	-0.0691
dstc11_banking_guard@lt	[0.0060, 0.0135]	1.0000	[1.0000, 1.0000]	0.9677	[0.8857, 1.0000]	0.9115	[0.8618, 0.9549]	-0.0563
dstc11_banking_guard@pl	[0.0044, 0.0182]	1.0000	[1.0000, 1.0000]	0.9804	[0.9259, 1.0000]	0.9080	[0.8277, 0.9633]	-0.0724
dstc11_banking_guard@sk	[0.0048, 0.0136]	1.0000	[1.0000, 1.0000]	0.9804	[0.9250, 1.0000]	0.9104	[0.8314, 0.9659]	-0.0700
dstc11_finance_guard	[0.0064, 0.0285]	0.98996	[0.9774, 1.0000]	0.8956	[0.8062, 0.9641]	0.9029	[0.8540, 0.9450]	0.0073
dstc11_finance_guard@de	[0.0156, 0.0644]	0.9758	[0.9444, 1.0000]	0.8545	[0.7619, 0.9365]	0.8617	[0.8030, 0.9168]	0.0072
dstc11_finance_guard@lt	[0.0117, 0.0587]	0.9639	[0.9299, 0.9917]	0.8313	[0.7400, 0.9157]	0.8382	[0.7836, 0.8917]	0.0069
dstc11_finance_guard@pl	[0.0159, 0.0653]	0.9504	[0.9037, 0.9864]	0.8085	[0.7069, 0.8968]	0.8400	[0.7878, 0.8917]	0.0315
dstc11_finance_guard@sk	[0.0175, 0.0677]	0.9669	[0.9298, 0.9940]	0.7815	[0.6710, 0.8766]	0.8309	[0.7702, 0.8874]	0.0494
dstc11_insurance	[0.0063, 0.0160]	0.9973	[0.9911, 1.0000]	0.9178	[0.7838, 0.9858]	0.9508	[0.9092, 0.9774]	0.0330
hint3_dev	[0.0038, 0.2910]	0.8806	[0.6920, 1.0000]	0.7463	[0.4938, 0.9592]	0.8759	[0.8023, 0.9753]	0.1296
ind_dev_unseen_electricity_grid_operator	[0.0255, 0.0622]	0.9728	[0.9381, 1.0000]	0.8639	[0.7801, 0.9408]	0.7727	[0.7173, 0.8326]	-0.0913
ind_dev_unseen_electricity_grid_operator@lt	[0.0325, 0.0845]	0.9406	[0.8876, 0.9839]	0.7426	[0.6216, 0.8571]	0.6374	[0.5670, 0.7056]	-0.1052
ind_dev_unseen_electricity_grid_operator@pl	[0.0209, 0.1010]	0.8947	[0.8406, 0.9460]	0.6842	[0.5672, 0.7931]	0.6372	[0.5872, 0.6901]	-0.0471
ind_dev_unseen_property_management_company	[0.0188, 0.0641]	0.9419	[0.8962, 0.9806]	0.8023	[0.7194, 0.8763]	0.7836	[0.7345, 0.8308]	-0.0187
ind_dev_unseen_property_management_company@lt	[0.0412, 0.1301]	0.8850	[0.8158, 0.9430]	0.6650	[0.5511, 0.7664]	0.6703	[0.6206, 0.7179]	0.0053
ind_dev_unseen_property_management_company@pl	[0.0283, 0.0986]	0.8859	[0.8204, 0.9449]	0.6413	[0.5163, 0.7627]	0.6652	[0.6152, 0.7187]	0.0239
ind_dev_unseen_tax_and_social_security_office	[0.0336, 0.0698]	0.9634	[0.9289, 0.9889]	0.8537	[0.7881, 0.9119]	0.7858	[0.7312, 0.8334]	-0.0679
ind_dev_unseen_tax_and_social_security_office@lt	[0.0737, 0.1184]	0.9746	[0.9480, 0.9956]	0.7766	[0.7022, 0.8504]	0.6630	[0.6063, 0.7188]	-0.1136
ind_dev_unseen_tax_and_social_security_office@pl	[0.0410, 0.0988]	0.9531	[0.9148, 0.9847]	0.8073	[0.7312, 0.8793]	0.6911	[0.6455, 0.7341]	-0.1162
ind_dev_unseen_telecom_operator	[0.0130, 0.0540]	0.9739	[0.9383, 1.0000]	0.7516	[0.6187, 0.8634]	0.7732	[0.7056, 0.8382]	0.0216
ind_dev_unseen_telecom_operator@lt	[0.0197, 0.0841]	0.9407	[0.8791, 0.9840]	0.6949	[0.5577, 0.8182]	0.7145	[0.6473, 0.7824]	0.0196
ind_dev_unseen_telecom_operator@pl	[0.0337, 0.1144]	0.9304	[0.8750, 0.9789]	0.6522	[0.4776, 0.8101]	0.7050	[0.6382, 0.7701]	0.0528
sgd_unseen	[0.0002, 0.0027]	1.0000	[1.0000, 1.0000]	0.8529	[0.5333, 1.0000]	0.9394	[0.8556, 0.9954]	0.0865
synth_dev_seen_domain	[0.0335, 0.0693]	0.9629	[0.9409, 0.9826]	0.8568	[0.7950, 0.9112]	0.8356	[0.7898, 0.8800]	-0.0211
synth_dev_seen_domain@de	[0.0254, 0.0688]	0.9465	[0.9124, 0.9778]	0.8495	[0.7833, 0.9097]	0.8184	[0.7671, 0.8676]	-0.0311
synth_dev_seen_domain@lt	[0.0326, 0.0767]	0.9281	[0.8949, 0.9590]	0.8007	[0.7305, 0.8669]	0.7604	[0.7018, 0.8180]	-0.0403
synth_dev_seen_domain@pl	[0.0443, 0.0915]	0.9381	[0.9012, 0.9706]	0.8282	[0.7632, 0.8881]	0.7779	[0.7221, 0.8320]	-0.0503
synth_dev_seen_domain@sk	[0.0519, 0.1075]	0.9463	[0.9146, 0.9764]	0.8356	[0.7741, 0.8964]	0.7594	[0.7017, 0.8147]	-0.0762
synth_dev_unseen_bookstore	n/a	1.0000	[1.0000, 1.0000]	0.9375	[0.9000, 1.0000]	0.9393	[0.9167, 0.9770]	0.0018
synth_dev_unseen_bookstore@de	n/a	1.0000	[1.0000, 1.0000]	1.0000	[1.0000, 1.0000]	0.9641	[0.9570, 0.9700]	-0.0359
synth_dev_unseen_bookstore@lt	n/a	1.0000	[1.0000, 1.0000]	0.9286	[0.8750, 1.0000]	0.8786	[0.8363, 0.9350]	-0.0500
synth_dev_unseen_bookstore@pl	n/a	1.0000	[1.0000, 1.0000]	1.0000	[1.0000, 1.0000]	0.9277	[0.8922, 0.9702]	-0.0723
synth_dev_unseen_bookstore@sk	n/a	1.0000	[1.0000, 1.0000]	1.0000	[1.0000, 1.0000]	0.9434	[0.9357, 0.9511]	-0.0566
synth_dev_unseen_camera_store	[0.0028, 0.0140]	1.0000	[1.0000, 1.0000]	0.8667	[0.8000, 0.9474]	0.7669	[0.6401, 0.8994]	-0.0998
synth_dev_unseen_car_dealership_service_department	[0.0088, 0.0248]	1.0000	[1.0000, 1.0000]	0.8947	[0.8333, 0.9474]	0.7933	[0.7361, 0.8497]	-0.1015
synth_dev_unseen_car_wash_subscription	[0.0168, 0.1198]	0.9452	[0.8548, 1.0000]	0.7123	[0.5652, 0.8594]	0.7666	[0.7181, 0.8097]	0.0543
synth_dev_unseen_dental_practice	[0.0289, 0.0966]	0.9527	[0.8889, 0.9933]	0.7770	[0.6875, 0.8600]	0.7199	[0.6705, 0.7688]	-0.0571
synth_dev_unseen_dental_practice@de	[0.0258, 0.0766]	0.9565	[0.8854, 1.0000]	0.7130	[0.5616, 0.8425]	0.6728	[0.6230, 0.7248]	-0.0403
synth_dev_unseen_dental_practice@lt	[0.0449, 0.1495]	0.9204	[0.8271, 1.0000]	0.6991	[0.5517, 0.8372]	0.6398	[0.5798, 0.6973]	-0.0594
synth_dev_unseen_dental_practice@pl	[0.0212, 0.0978]	0.8974	[0.8197, 0.9620]	0.6667	[0.5455, 0.7848]	0.6179	[0.5631, 0.6711]	-0.0488
synth_dev_unseen_dental_practice@sk	[0.0283, 0.1114]	0.9138	[0.8239, 0.9839]	0.6293	[0.5139, 0.7429]	0.6360	[0.5887, 0.6747]	0.0066
synth_dev_unseen_ebook_platform	[0.0175, 0.0442]	1.0000	[1.0000, 1.0000]	0.8594	[0.7788, 0.9348]	0.8366	[0.7742, 0.8960]	-0.0227
synth_dev_unseen_ebook_platform@de	[0.0123, 0.0437]	0.9902	[0.9677, 1.0000]	0.8333	[0.7209, 0.9276]	0.8048	[0.7260, 0.8764]	-0.0285
synth_dev_unseen_ebook_platform@lt	[0.0228, 0.0594]	0.9904	[0.9688, 1.0000]	0.8077	[0.6805, 0.9000]	0.7700	[0.6979, 0.8336]	-0.0377
synth_dev_unseen_ebook_platform@pl	[0.0111, 0.1119]	0.9300	[0.8359, 1.0000]	0.8000	[0.6935, 0.8947]	0.7599	[0.6881, 0.8288]	-0.0401
synth_dev_unseen_ebook_platform@sk	[0.0265, 0.0774]	0.9700	[0.9362, 1.0000]	0.8000	[0.6456, 0.9173]	0.7825	[0.7275, 0.8408]	-0.0175
synth_dev_unseen_home_appliance_repair	[0.0088, 0.0405]	0.9877	[0.9592, 1.0000]	0.8642	[0.7666, 0.9500]	0.8212	[0.7552, 0.8823]	-0.0430
synth_dev_unseen_home_appliance_repair@de	[0.0094, 0.0944]	0.9552	[0.8857, 1.0000]	0.8209	[0.7121, 0.9231]	0.7841	[0.7134, 0.8513]	-0.0368
synth_dev_unseen_home_appliance_repair@lt	[0.0129, 0.1078]	0.9242	[0.8604, 0.9822]	0.7273	[0.6250, 0.8432]	0.7230	[0.6721, 0.7698]	-0.0043
synth_dev_unseen_home_appliance_repair@pl	[0.0174, 0.0962]	0.9524	[0.8768, 1.0000]	0.7460	[0.6628, 0.8429]	0.7248	[0.6624, 0.7886]	-0.0213
synth_dev_unseen_home_appliance_repair@sk	[0.0226, 0.1457]	0.9403	[0.8399, 1.0000]	0.7612	[0.6744, 0.8589]	0.7307	[0.6783, 0.7853]	-0.0305
synth_dev_unseen_online_auction_site	[0.0132, 0.0434]	0.9829	[0.9565, 1.0000]	0.9060	[0.8333, 0.9685]	0.8460	[0.7925, 0.8992]	-0.0600
synth_dev_unseen_online_auction_site@de	[0.0169, 0.0480]	0.9880	[0.9592, 1.0000]	0.8795	[0.7538, 0.9747]	0.8349	[0.7688, 0.8962]	-0.0447
synth_dev_unseen_online_auction_site@lt	[0.0311, 0.0584]	1.0000	[1.0000, 1.0000]	0.8602	[0.7500, 0.9575]	0.7633	[0.6933, 0.8266]	-0.0969
synth_dev_unseen_online_auction_site@pl	[0.0098, 0.0828]	0.9545	[0.9024, 1.0000]	0.7955	[0.7045, 0.8913]	0.7357	[0.6819, 0.7892]	-0.0598
synth_dev_unseen_online_auction_site@sk	[0.0120, 0.0680]	0.9556	[0.9157, 0.9897]	0.8222	[0.7424, 0.9000]	0.7410	[0.6940, 0.7881]	-0.0812
synth_dev_unseen_student_services_office	[0.0216, 0.0507]	0.9774	[0.9485, 1.0000]	0.8927	[0.8143, 0.9600]	0.8559	[0.8010, 0.9069]	-0.0367
synth_dev_unseen_tattoo_studio	[0.0178, 0.1325]	0.9123	[0.8378, 0.9722]	0.6842	[0.4903, 0.8421]	0.6190	[0.5335, 0.7129]	-0.0652
synth_dev_unseen_tattoo_studio@de	[0.0412, 0.3105]	0.8256	[0.6517, 0.9649]	0.6860	[0.4773, 0.8571]	0.5948	[0.5139, 0.6803]	-0.0912
synth_dev_unseen_tattoo_studio@lt	[0.0294, 0.2907]	0.7931	[0.6116, 0.9310]	0.6322	[0.4235, 0.8182]	0.5454	[0.4596, 0.6386]	-0.0868
synth_dev_unseen_tattoo_studio@pl	[0.0242, 0.2841]	0.8280	[0.6547, 0.9615]	0.6022	[0.4203, 0.7647]	0.5504	[0.4800, 0.6292]	-0.0518
synth_dev_unseen_tattoo_studio@sk	[0.0287, 0.2522]	0.8222	[0.6590, 0.9444]	0.6222	[0.3833, 0.8371]	0.5467	[0.4685, 0.6295]	-0.0755
synth_dev_unseen_vpn_service	[0.0077, 0.0262]	1.0000	[1.0000, 1.0000]	0.9231	[0.8269, 1.0000]	0.8459	[0.7857, 0.9150]	-0.0772
synth_dev_unseen_vpn_service@de	[0.0070, 0.0400]	1.0000	[1.0000, 1.0000]	0.9423	[0.8413, 1.0000]	0.8232	[0.7277, 0.9161]	-0.1191
synth_dev_unseen_vpn_service@lt	[0.0118, 0.0479]	1.0000	[1.0000, 1.0000]	0.8958	[0.8000, 0.9730]	0.7837	[0.7020, 0.8749]	-0.1121
synth_dev_unseen_vpn_service@pl	[0.0118, 0.0432]	1.0000	[1.0000, 1.0000]	0.9184	[0.8157, 1.0000]	0.7865	[0.6895, 0.8798]	-0.1319
synth_dev_unseen_vpn_service@sk	[0.0125, 0.0814]	0.9800	[0.9286, 1.0000]	0.8400	[0.6469, 1.0000]	0.7797	[0.6529, 0.8914]	-0.0603
synth_dev_unseen_waste_collection_service	[0.0367, 0.1455]	0.8631	[0.7558, 0.9551]	0.6071	[0.4540, 0.7695]	0.6045	[0.5274, 0.6903]	-0.0026
synth_dev_unseen_yoga_studio	[0.0194, 0.1198]	0.8864	[0.8033, 0.9588]	0.6439	[0.5180, 0.7569]	0.6497	[0.5831, 0.7091]	0.0058
synth_dev_unseen_yoga_studio@de	[0.0198, 0.1430]	0.8972	[0.7971, 0.9733]	0.5888	[0.4868, 0.6825]	0.6281	[0.5729, 0.6786]	0.0393
synth_dev_unseen_yoga_studio@lt	[0.0270, 0.1354]	0.8558	[0.7851, 0.9262]	0.5577	[0.4426, 0.6767]	0.5818	[0.5232, 0.6311]	0.0241
synth_dev_unseen_yoga_studio@pl	[0.0339, 0.2544]	0.7921	[0.6438, 0.9121]	0.5644	[0.4144, 0.7093]	0.5896	[0.5318, 0.6430]	0.0253
synth_dev_unseen_yoga_studio@sk	[0.0362, 0.1814]	0.8218	[0.7088, 0.9231]	0.5347	[0.4000, 0.6860]	0.5778	[0.5133, 0.6475]	0.0432

information.by_pool	scores.mean_top_minus_accuracy_interval	scores.brier	scores.brier_interval	check.pairs	check.ece	check.ba	check.tyr	check.tnr
abcd_dev	[0.0297, 0.1671]	0.5341	[0.4288, 0.6462]	2,520	0.0752	0.8321	0.7381	0.9261
abcd_dev@de	[0.0058, 0.1803]	0.5985	[0.4817, 0.7257]	1,300	0.1057	0.7841	0.6615	0.9067
abcd_dev@lt	[0.0058, 0.1824]	0.6190	[0.5033, 0.7395]	1,325	0.0958	0.7925	0.6906	0.8943
abcd_dev@pl	[-0.0060, 0.1810]	0.6040	[0.4895, 0.7243]	1,215	0.1187	0.7814	0.6379	0.9249
abcd_dev@sk	[0.0088, 0.1520]	0.5849	[0.4702, 0.7027]	1,265	0.1132	0.7831	0.6443	0.9219
bitext_dev	[-0.0970, 0.0138]	0.1362	[0.0417, 0.2614]	2,540	0.0510	0.9473	0.9213	0.9734
clinc_dev@de	[-0.0919, 0.0382]	0.2671	[0.1690, 0.3742]	2,320	0.1121	0.8901	0.7909	0.9892
clinc_dev@lt	[-0.0794, 0.0369]	0.3366	[0.2433, 0.4385]	2,430	0.1377	0.8678	0.7469	0.9887
clinc_dev@pl	[-0.0755, 0.0250]	0.3128	[0.2274, 0.4084]	2,395	0.1303	0.8719	0.7557	0.9880
clinc_dev@sk	[-0.0685, 0.0410]	0.3394	[0.2383, 0.4428]	2,410	0.1326	0.8651	0.7407	0.9896
dstc11_banking_guard	[-0.0620, -0.0167]	0.0284	[0.0114, 0.0503]	835	0.0557	0.9603	0.9940	0.9266
dstc11_banking_guard@de	[-0.1138, -0.0239]	0.0707	[0.0081, 0.1450]	325	0.0603	0.9635	1.0000	0.9269
dstc11_banking_guard@lt	[-0.0964, -0.0106]	0.0596	[0.0087, 0.1616]	310	0.0641	0.9315	0.9839	0.8790
dstc11_banking_guard@pl	[-0.1075, -0.0367]	0.0499	[0.0066, 0.1287]	255	0.0694	0.9265	0.9412	0.9118
dstc11_banking_guard@sk	[-0.1023, -0.0334]	0.0593	[0.0042, 0.1686]	255	0.0602	0.9412	0.9608	0.9216
dstc11_finance_guard	[-0.0437, 0.0724]	0.1591	[0.0616, 0.2784]	2,490	0.0330	0.9558	0.9458	0.9659
dstc11_finance_guard@de	[-0.0538, 0.0783]	0.2192	[0.1098, 0.3463]	825	0.0503	0.9477	0.9273	0.9682
dstc11_finance_guard@lt	[-0.0616, 0.0810]	0.2431	[0.1308, 0.3682]	830	0.0489	0.9450	0.9217	0.9684
dstc11_finance_guard@pl	[-0.0259, 0.1040]	0.2743	[0.1558, 0.4172]	705	0.0565	0.9211	0.8723	0.9699
dstc11_finance_guard@sk	[-0.0176, 0.1215]	0.3163	[0.1819, 0.4716]	755	0.0605	0.9123	0.8543	0.9702
dstc11_insurance	[-0.0197, 0.1270]	0.1144	[0.0269, 0.2798]	1,885	0.0834	0.9456	0.9920	0.8992
hint3_dev	[-0.0361, 0.3446]	0.4070	[0.0768, 0.7819]	335	0.1752	0.8004	0.6119	0.9888
ind_dev_unseen_electricity_grid_operator	[-0.1608, -0.0203]	0.2300	[0.1357, 0.3284]	735	0.1122	0.8707	0.9592	0.7823
ind_dev_unseen_electricity_grid_operator@lt	[-0.1980, -0.0109]	0.3730	[0.2607, 0.4936]	505	0.0917	0.8428	0.9406	0.7450
ind_dev_unseen_electricity_grid_operator@pl	[-0.1359, 0.0513]	0.4640	[0.3418, 0.5963]	475	0.0773	0.8237	0.8316	0.8158
ind_dev_unseen_property_management_company	[-0.0690, 0.0352]	0.2550	[0.1774, 0.3415]	1,290	0.0509	0.9109	0.9419	0.8798
ind_dev_unseen_property_management_company@lt	[-0.0771, 0.0938]	0.4206	[0.3161, 0.5439]	1,000	0.0332	0.8681	0.8600	0.8762
ind_dev_unseen_property_management_company@pl	[-0.0701, 0.1310]	0.4504	[0.3381, 0.5700]	920	0.0688	0.8879	0.8533	0.9226
ind_dev_unseen_tax_and_social_security_office	[-0.1191, -0.0209]	0.2022	[0.1409, 0.2744]	1,230	0.0983	0.8847	0.9675	0.8018
ind_dev_unseen_tax_and_social_security_office@lt	[-0.1811, -0.0469]	0.3538	[0.2676, 0.4362]	985	0.0767	0.8858	0.9543	0.8173
ind_dev_unseen_tax_and_social_security_office@pl	[-0.1744, -0.0591]	0.2994	[0.2241, 0.3798]	960	0.0838	0.8893	0.9635	0.8151
ind_dev_unseen_telecom_operator	[-0.0747, 0.1284]	0.3382	[0.2053, 0.4893]	765	0.0511	0.8995	0.8824	0.9167
ind_dev_unseen_telecom_operator@lt	[-0.0689, 0.1314]	0.4354	[0.2678, 0.6182]	590	0.0645	0.8729	0.8305	0.9153
ind_dev_unseen_telecom_operator@pl	[-0.0713, 0.2005]	0.4756	[0.2871, 0.6837]	575	0.0829	0.8609	0.8087	0.9130
sgd_unseen	[-0.0198, 0.3223]	0.2496	[0.0006, 0.8005]	408	0.0632	0.9559	0.9118	1.0000
synth_dev_seen_domain	[-0.0594, 0.0172]	0.1846	[0.1232, 0.2512]	1,885	0.0537	0.9456	0.9098	0.9814
synth_dev_seen_domain@de	[-0.0681, 0.0056]	0.1974	[0.1295, 0.2640]	1,495	0.0549	0.9536	0.9231	0.9841
synth_dev_seen_domain@lt	[-0.0778, -0.0011]	0.2513	[0.1788, 0.3254]	1,530	0.0597	0.9285	0.8758	0.9812
synth_dev_seen_domain@pl	[-0.0886, -0.0114]	0.2468	[0.1743, 0.3207]	1,455	0.0564	0.9381	0.8969	0.9794
synth_dev_seen_domain@sk	[-0.1112, -0.0419]	0.2516	[0.1743, 0.3283]	1,490	0.0747	0.9274	0.8826	0.9723
synth_dev_unseen_bookstore	[-0.0230, 0.0167]	0.0679	[0.0020, 0.1073]	32	0.3174	0.5938	1.0000	0.1875
synth_dev_unseen_bookstore@de	[-0.0430, -0.0300]	0.0042	[0.0031, 0.0055]	22	0.3648	0.6364	0.9091	0.3636
synth_dev_unseen_bookstore@lt	[-0.0650, -0.0387]	0.1091	[0.0209, 0.1753]	28	0.3602	0.5714	0.9286	0.2143
synth_dev_unseen_bookstore@pl	[-0.1078, -0.0298]	0.0339	[0.0023, 0.0602]	22	0.3241	0.6364	0.9091	0.3636
synth_dev_unseen_bookstore@sk	[-0.0643, -0.0489]	0.0116	[0.0113, 0.0118]	24	0.3511	0.5833	0.9167	0.2500
synth_dev_unseen_camera_store	[-0.1599, -0.0420]	0.2657	[0.0956, 0.4133]	225	0.1308	0.7472	0.9111	0.5833
synth_dev_unseen_car_dealership_service_department	[-0.1896, 0.0029]	0.1770	[0.1359, 0.2242]	285	0.0913	0.8618	0.8947	0.8289
synth_dev_unseen_car_wash_subscription	[-0.0655, 0.1935]	0.4031	[0.2069, 0.6202]	365	0.0814	0.8288	0.8904	0.7671
synth_dev_unseen_dental_practice	[-0.1149, 0.0038]	0.3484	[0.2314, 0.4736]	740	0.0594	0.8775	0.9257	0.8294
synth_dev_unseen_dental_practice@de	[-0.1458, 0.0797]	0.3987	[0.2597, 0.5492]	575	0.0785	0.8717	0.9130	0.8304
synth_dev_unseen_dental_practice@lt	[-0.1678, 0.0531]	0.4259	[0.2844, 0.5788]	565	0.0714	0.8263	0.8142	0.8385
synth_dev_unseen_dental_practice@pl	[-0.1493, 0.0447]	0.4712	[0.3285, 0.6222]	585	0.0582	0.8248	0.8120	0.8376
synth_dev_unseen_dental_practice@sk	[-0.0872, 0.0995]	0.5052	[0.3736, 0.6412]	580	0.0616	0.8416	0.8534	0.8297
synth_dev_unseen_ebook_platform	[-0.0754, 0.0336]	0.1874	[0.0972, 0.2916]	640	0.1191	0.8750	0.9844	0.7656
synth_dev_unseen_ebook_platform@de	[-0.0819, 0.0304]	0.2286	[0.1083, 0.3864]	510	0.0901	0.8971	0.9608	0.8333
synth_dev_unseen_ebook_platform@lt	[-0.1067, 0.0405]	0.2651	[0.1524, 0.4072]	520	0.0934	0.8726	0.9519	0.7933
synth_dev_unseen_ebook_platform@pl	[-0.1242, 0.0491]	0.3047	[0.1785, 0.4550]	500	0.0677	0.8575	0.8900	0.8250
synth_dev_unseen_ebook_platform@sk	[-0.1267, 0.1148]	0.2952	[0.1618, 0.4805]	500	0.0903	0.9012	0.9600	0.8425
synth_dev_unseen_home_appliance_repair	[-0.0944, 0.0098]	0.1619	[0.0701, 0.2771]	405	0.0827	0.9167	0.9259	0.9074
synth_dev_unseen_home_appliance_repair@de	[-0.1149, 0.0434]	0.2580	[0.1263, 0.3858]	335	0.0788	0.8974	0.9104	0.8843
synth_dev_unseen_home_appliance_repair@lt	[-0.1086, 0.0865]	0.3677	[0.2478, 0.4733]	330	0.0909	0.8314	0.8030	0.8598
synth_dev_unseen_home_appliance_repair@pl	[-0.0695, 0.0318]	0.3147	[0.2150, 0.3927]	315	0.0671	0.8829	0.8889	0.8770
synth_dev_unseen_home_appliance_repair@sk	[-0.1249, 0.0521]	0.3224	[0.2217, 0.4148]	335	0.0949	0.8601	0.8507	0.8694
synth_dev_unseen_online_auction_site	[-0.1042, -0.0186]	0.1439	[0.0686, 0.2286]	585	0.1004	0.8590	0.9402	0.7778
synth_dev_unseen_online_auction_site@de	[-0.1030, 0.0264]	0.1769	[0.0711, 0.3035]	415	0.0704	0.9051	0.9639	0.8464
synth_dev_unseen_online_auction_site@lt	[-0.1782, -0.0103]	0.2432	[0.1283, 0.3698]	465	0.0647	0.8495	0.8817	0.8172
synth_dev_unseen_online_auction_site@pl	[-0.1327, 0.0134]	0.2897	[0.1901, 0.3922]	440	0.0667	0.8324	0.8750	0.7898
synth_dev_unseen_online_auction_site@sk	[-0.1416, -0.0274]	0.2468	[0.1684, 0.3285]	450	0.0966	0.8611	0.9000	0.8222
synth_dev_unseen_student_services_office	[-0.0788, 0.0190]	0.1620	[0.0691, 0.2693]	885	0.0460	0.9167	0.9379	0.8955
synth_dev_unseen_tattoo_studio	[-0.2112, 0.1522]	0.4985	[0.2947, 0.7620]	570	0.1549	0.7050	0.8421	0.5680
synth_dev_unseen_tattoo_studio@de	[-0.2314, 0.0866]	0.4875	[0.2864, 0.7369]	430	0.1119	0.7209	0.7674	0.6744
synth_dev_unseen_tattoo_studio@lt	[-0.2502, 0.0968]	0.5394	[0.3389, 0.7662]	435	0.1258	0.7083	0.7586	0.6580
synth_dev_unseen_tattoo_studio@pl	[-0.1814, 0.1084]	0.5646	[0.3803, 0.7839]	465	0.1296	0.6949	0.7419	0.6478
synth_dev_unseen_tattoo_studio@sk	[-0.2565, 0.1300]	0.5463	[0.3565, 0.7618]	450	0.1177	0.6958	0.6778	0.7139
synth_dev_unseen_vpn_service	[-0.1834, 0.0511]	0.1544	[0.0646, 0.3091]	325	0.0657	0.8577	0.9077	0.8077
synth_dev_unseen_vpn_service@de	[-0.2300, 0.0252]	0.1783	[0.0533, 0.3595]	260	0.0652	0.8341	0.8077	0.8606
synth_dev_unseen_vpn_service@lt	[-0.2071, 0.0394]	0.2308	[0.1040, 0.4008]	240	0.1221	0.8021	0.7083	0.8958
synth_dev_unseen_vpn_service@pl	[-0.2538, 0.0265]	0.2006	[0.0866, 0.3765]	245	0.0875	0.8138	0.7347	0.8929
synth_dev_unseen_vpn_service@sk	[-0.2000, 0.1048]	0.2271	[0.0623, 0.4151]	250	0.1177	0.8350	0.7800	0.8900
synth_dev_unseen_waste_collection_service	[-0.1127, 0.1043]	0.4776	[0.3253, 0.6229]	840	0.1695	0.7418	0.9167	0.5670
synth_dev_unseen_yoga_studio	[-0.0943, 0.1152]	0.4870	[0.3718, 0.6193]	660	0.0817	0.8106	0.8712	0.7500
synth_dev_unseen_yoga_studio@de	[-0.0389, 0.1152]	0.5153	[0.3963, 0.6364]	535	0.0789	0.8154	0.8505	0.7804
synth_dev_unseen_yoga_studio@lt	[-0.0882, 0.1450]	0.5784	[0.4597, 0.7048]	520	0.0778	0.7764	0.7788	0.7740
synth_dev_unseen_yoga_studio@pl	[-0.0815, 0.1254]	0.5337	[0.4040, 0.6611]	505	0.0707	0.7933	0.8218	0.7649
synth_dev_unseen_yoga_studio@sk	[-0.0666, 0.1538]	0.6014	[0.4625, 0.7197]	505	0.0704	0.7822	0.7228	0.8416

information.by_pool	check.brier	check.auc	check.right_yes_0.5	check.right_yes_0.2	check.right_yes_0.05	check.ece_p975_twoway	check.ba_p025_twoway	check.ece_interval_twoway
abcd_dev	0.1267	0.9028	0.9090	0.7140	0.3445	0.1396	0.7791	[0.0445, 0.1396]
abcd_dev@de	0.1519	0.8784	0.8764	0.6394	0.2718	0.1796	0.7245	[0.0721, 0.1796]
abcd_dev@lt	0.1536	0.8723	0.8673	0.6203	0.2559	0.1723	0.7361	[0.0634, 0.1723]
abcd_dev@pl	0.1620	0.8657	0.8947	0.6798	0.3089	0.1995	0.7154	[0.0848, 0.1995]
abcd_dev@sk	0.1522	0.8803	0.8919	0.6736	0.3028	0.1850	0.7284	[0.0780, 0.1850]
bitext_dev	0.0358	0.9915	0.9720	0.8966	0.6460	0.1086	0.8822	[0.0351, 0.1086]
clinc_dev@de	0.0796	0.9853	0.9866	0.9483	0.7944	0.1657	0.8307	[0.0802, 0.1657]
clinc_dev@lt	0.1044	0.9764	0.9851	0.9429	0.7765	0.1946	0.8087	[0.0997, 0.1946]
clinc_dev@pl	0.0981	0.9775	0.9844	0.9403	0.7682	0.1853	0.8171	[0.0920, 0.1853]
clinc_dev@sk	0.1000	0.9759	0.9862	0.9469	0.7898	0.1912	0.8037	[0.0937, 0.1912]
dstc11_banking_guard	0.0365	0.9911	0.9313	0.7721	0.4163	0.1033	0.9045	[0.0514, 0.1033]
dstc11_banking_guard@de	0.0386	0.9885	0.9319	0.7738	0.4187	0.1219	0.8880	[0.0545, 0.1219]
dstc11_banking_guard@lt	0.0529	0.9792	0.8905	0.6703	0.2998	0.1441	0.8384	[0.0616, 0.1441]
dstc11_banking_guard@pl	0.0488	0.9839	0.9143	0.7273	0.3596	0.1469	0.8513	[0.0636, 0.1469]
dstc11_banking_guard@sk	0.0473	0.9826	0.9245	0.7538	0.3920	0.1335	0.8685	[0.0607, 0.1335]
dstc11_finance_guard	0.0342	0.99004	0.9652	0.8738	0.5932	0.0589	0.9256	[0.0290, 0.0589]
dstc11_finance_guard@de	0.0444	0.9857	0.9668	0.8793	0.6053	0.0997	0.9132	[0.0427, 0.0997]
dstc11_finance_guard@lt	0.0502	0.9790	0.9668	0.8793	0.6053	0.1005	0.9092	[0.0436, 0.1005]
dstc11_finance_guard@pl	0.0641	0.9713	0.9666	0.8786	0.6037	0.1218	0.8703	[0.0435, 0.1218]
dstc11_finance_guard@sk	0.0655	0.9721	0.9663	0.8776	0.6014	0.1278	0.8559	[0.0453, 0.1278]
dstc11_insurance	0.0436	0.9945	0.9078	0.7110	0.3412	0.1213	0.9004	[0.0497, 0.1213]
hint3_dev	0.1553	0.8766	0.9820	0.9318	0.7421	n/a	n/a	[0.0687, 0.4008]
ind_dev_unseen_electricity_grid_operator	0.0946	0.9682	0.8150	0.5242	0.1883	0.1662	0.8223	[0.0709, 0.1662]
ind_dev_unseen_electricity_grid_operator@lt	0.1146	0.9329	0.7867	0.4798	0.1626	0.1595	0.7878	[0.0774, 0.1595]
ind_dev_unseen_electricity_grid_operator@pl	0.1153	0.9227	0.8187	0.5302	0.1920	0.1539	0.7613	[0.0694, 0.1539]
ind_dev_unseen_property_management_company	0.0701	0.9710	0.8869	0.6621	0.2921	0.0887	0.8780	[0.0398, 0.0887]
ind_dev_unseen_property_management_company@lt	0.0961	0.9426	0.8742	0.6347	0.2678	0.0965	0.8210	[0.0398, 0.0965]
ind_dev_unseen_property_management_company@pl	0.0827	0.9613	0.9168	0.7336	0.3670	0.1223	0.8428	[0.0577, 0.1223]
ind_dev_unseen_tax_and_social_security_office	0.0796	0.9779	0.8300	0.5497	0.2044	0.1325	0.8516	[0.0627, 0.1325]
ind_dev_unseen_tax_and_social_security_office@lt	0.0836	0.9637	0.8393	0.5663	0.2156	0.1133	0.8490	[0.0513, 0.1133]
ind_dev_unseen_tax_and_social_security_office@pl	0.0811	0.9706	0.8390	0.5657	0.2152	0.1194	0.8578	[0.0572, 0.1194]
ind_dev_unseen_telecom_operator	0.0736	0.9646	0.9137	0.7258	0.3579	0.1095	0.8483	[0.0412, 0.1095]
ind_dev_unseen_telecom_operator@lt	0.0961	0.9436	0.9074	0.7101	0.3403	0.1484	0.8117	[0.0562, 0.1484]
ind_dev_unseen_telecom_operator@pl	0.1010	0.9372	0.9029	0.6992	0.3286	0.1797	0.7725	[0.0690, 0.1797]
sgd_unseen	0.0344	0.9948	1.0000	1.0000	1.0000	n/a	n/a	[0.0369, 0.2205]
synth_dev_seen_domain	0.0403	0.9924	0.9800	0.9245	0.7206	0.0805	0.9206	[0.0437, 0.0805]
synth_dev_seen_domain@de	0.0419	0.9903	0.9831	0.9356	0.7536	0.0808	0.9292	[0.0478, 0.0808]
synth_dev_seen_domain@lt	0.0532	0.9865	0.9790	0.9210	0.7104	0.0946	0.8943	[0.049998, 0.0946]
synth_dev_seen_domain@pl	0.0548	0.9835	0.9775	0.9158	0.6960	0.0914	0.9090	[0.0465, 0.0914]
synth_dev_seen_domain@sk	0.0535	0.9868	0.9696	0.8885	0.6266	0.1048	0.8946	[0.0604, 0.1048]
synth_dev_unseen_bookstore	0.2763	0.8867	0.5517	0.2353	0.0608	n/a	n/a	[0.0361, 0.4771]
synth_dev_unseen_bookstore@de	0.2810	0.9091	0.5882	0.2632	0.0699	n/a	n/a	[0.0366, 0.4829]
synth_dev_unseen_bookstore@lt	0.3078	0.8010	0.5417	0.2281	0.0586	n/a	n/a	[0.0546, 0.5086]
synth_dev_unseen_bookstore@pl	0.2767	0.8430	0.5882	0.2632	0.0699	n/a	n/a	[0.0908, 0.4545]
synth_dev_unseen_bookstore@sk	0.3189	0.8750	0.5500	0.2340	0.0604	n/a	n/a	[0.03002, 0.4842]
synth_dev_unseen_camera_store	0.1683	0.8607	0.6862	0.3534	0.1032	n/a	n/a	[0.0715, 0.2665]
synth_dev_unseen_car_dealership_service_department	0.0929	0.9538	0.8395	0.5667	0.2159	n/a	n/a	[0.0776, 0.2065]
synth_dev_unseen_car_wash_subscription	0.1118	0.9304	0.7927	0.4887	0.1675	0.1750	0.7655	[0.0731, 0.1750]
synth_dev_unseen_dental_practice	0.0863	0.9592	0.8444	0.5756	0.2221	0.1163	0.8185	[0.0545, 0.1163]
synth_dev_unseen_dental_practice@de	0.0900	0.9541	0.8434	0.5738	0.2208	0.1390	0.8043	[0.0704, 0.1390]
synth_dev_unseen_dental_practice@lt	0.1128	0.9211	0.8345	0.5576	0.2097	0.1657	0.7453	[0.0682, 0.1657]
synth_dev_unseen_dental_practice@pl	0.1111	0.9249	0.8333	0.5556	0.2083	0.1439	0.7414	[0.0594, 0.1439]
synth_dev_unseen_dental_practice@sk	0.1101	0.9263	0.8337	0.5562	0.2088	0.1553	0.7575	[0.0590, 0.1553]
synth_dev_unseen_ebook_platform	0.0960	0.9677	0.8077	0.5122	0.1810	0.1928	0.8008	[0.0629, 0.1928]
synth_dev_unseen_ebook_platform@de	0.0756	0.9772	0.8522	0.5904	0.2328	0.1444	0.8474	[0.0672, 0.1444]
synth_dev_unseen_ebook_platform@lt	0.0956	0.9592	0.8216	0.5351	0.1951	0.1570	0.8117	[0.0638, 0.1570]
synth_dev_unseen_ebook_platform@pl	0.1004	0.9443	0.8357	0.5597	0.2112	0.1355	0.8107	[0.0633, 0.1355]
synth_dev_unseen_ebook_platform@sk	0.0802	0.9683	0.8591	0.6038	0.2429	0.1426	0.8555	[0.0750, 0.1426]
synth_dev_unseen_home_appliance_repair	0.0625	0.9779	0.9091	0.7143	0.3448	n/a	n/a	[0.0713, 0.1399]
synth_dev_unseen_home_appliance_repair@de	0.0794	0.9620	0.8873	0.6630	0.2929	n/a	n/a	[0.0752, 0.1663]
synth_dev_unseen_home_appliance_repair@lt	0.1170	0.9166	0.8514	0.5889	0.2317	n/a	n/a	[0.0788, 0.2018]
synth_dev_unseen_home_appliance_repair@pl	0.0935	0.9458	0.8784	0.6437	0.2755	n/a	n/a	[0.0672, 0.1760]
synth_dev_unseen_home_appliance_repair@sk	0.1029	0.9294	0.8669	0.6196	0.2553	n/a	n/a	[0.0868, 0.1969]
synth_dev_unseen_online_auction_site	0.1025	0.9536	0.8088	0.5140	0.1821	0.1838	0.7837	[0.0594, 0.1838]
synth_dev_unseen_online_auction_site@de	0.0738	0.9712	0.8625	0.6107	0.2483	0.1382	0.8640	[0.0681, 0.1382]
synth_dev_unseen_online_auction_site@lt	0.1025	0.9440	0.8283	0.5467	0.2025	0.1442	0.7906	[0.0589, 0.1442]
synth_dev_unseen_online_auction_site@pl	0.1112	0.9280	0.8063	0.5099	0.1797	0.1600	0.7707	[0.0686, 0.1600]
synth_dev_unseen_online_auction_site@sk	0.1076	0.9346	0.8351	0.5586	0.2104	0.1633	0.8101	[0.0889, 0.1633]
synth_dev_unseen_student_services_office	0.0568	0.9817	0.8997	0.6917	0.3208	0.0903	0.8796	[0.0432, 0.0903]
synth_dev_unseen_tattoo_studio	0.1958	0.8324	0.6609	0.3276	0.0930	0.2481	0.6396	[0.1048, 0.2481]
synth_dev_unseen_tattoo_studio@de	0.1861	0.8192	0.7021	0.3708	0.1104	0.2101	0.6553	[0.0972, 0.2101]
synth_dev_unseen_tattoo_studio@lt	0.1959	0.7987	0.6893	0.3568	0.1046	0.2217	0.6457	[0.1137, 0.2217]
synth_dev_unseen_tattoo_studio@pl	0.2004	0.7922	0.6781	0.3450	0.0998	0.2268	0.6436	[0.1173, 0.2268]
synth_dev_unseen_tattoo_studio@sk	0.1987	0.7915	0.7032	0.3720	0.1109	0.2355	0.6303	[0.1006, 0.2355]
synth_dev_unseen_vpn_service	0.1001	0.9410	0.8252	0.5413	0.1990	n/a	n/a	[0.0654, 0.1960]
synth_dev_unseen_vpn_service@de	0.1085	0.9246	0.8528	0.5915	0.2337	n/a	n/a	[0.0591, 0.2629]
synth_dev_unseen_vpn_service@lt	0.1252	0.9209	0.8718	0.6296	0.2636	n/a	n/a	[0.0761, 0.3474]
synth_dev_unseen_vpn_service@pl	0.1114	0.9261	0.8727	0.6316	0.2652	n/a	n/a	[0.0792, 0.2784]
synth_dev_unseen_vpn_service@sk	0.1255	0.8933	0.8764	0.6393	0.2718	n/a	n/a	[0.0937, 0.3290]
synth_dev_unseen_waste_collection_service	0.1833	0.8782	0.6792	0.3461	0.1002	0.2453	0.6727	[0.1035, 0.2453]
synth_dev_unseen_yoga_studio	0.1334	0.8984	0.7770	0.4656	0.1550	0.1620	0.7445	[0.0703, 0.1620]
synth_dev_unseen_yoga_studio@de	0.1347	0.8873	0.7948	0.4919	0.1693	0.1688	0.7516	[0.0707, 0.1688]
synth_dev_unseen_yoga_studio@lt	0.1554	0.8543	0.7751	0.4629	0.1536	0.1782	0.7054	[0.0724, 0.1782]
synth_dev_unseen_yoga_studio@pl	0.1504	0.8651	0.7775	0.4663	0.1554	0.1833	0.7113	[0.0619, 0.1833]
synth_dev_unseen_yoga_studio@sk	0.1501	0.8658	0.8202	0.5328	0.1936	0.1940	0.7088	[0.0574, 0.1940]

information.by_pool	check.ba_interval_twoway	check.tyr_interval_twoway	check.tnr_interval_twoway	check.brier_interval_twoway	check.auc_interval_twoway	check.right_yes_0.5_interval_twoway	check.right_yes_0.2_interval_twoway	check.right_yes_0.05_interval_twoway
abcd_dev	[0.7791, 0.8703]	[0.6558, 0.8129]	[0.8843, 0.9567]	[0.1024, 0.1603]	[0.8720, 0.9300]	[0.8657, 0.9444]	[0.6171, 0.8093]	[0.2534, 0.4719]
abcd_dev@de	[0.7245, 0.8301]	[0.5683, 0.7500]	[0.8545, 0.9466]	[0.1226, 0.1911]	[0.8418, 0.9135]	[0.8200, 0.9246]	[0.5325, 0.7541]	[0.1934, 0.3923]
abcd_dev@lt	[0.7361, 0.8357]	[0.6042, 0.7740]	[0.8383, 0.9375]	[0.1261, 0.1925]	[0.8315, 0.9088]	[0.8112, 0.9183]	[0.5178, 0.7374]	[0.1844, 0.3716]
abcd_dev@pl	[0.7154, 0.8283]	[0.5451, 0.7303]	[0.8808, 0.9579]	[0.1320, 0.2034]	[0.8274, 0.9016]	[0.8423, 0.9380]	[0.5718, 0.7910]	[0.2194, 0.4434]
abcd_dev@sk	[0.7284, 0.8230]	[0.5637, 0.7199]	[0.8780, 0.9556]	[0.1257, 0.1891]	[0.8431, 0.9154]	[0.8414, 0.9362]	[0.5702, 0.7857]	[0.2183, 0.4356]
bitext_dev	[0.8822, 0.9910]	[0.8043, 0.9970]	[0.9372, 0.9984]	[0.0117, 0.0680]	[0.9774, 0.9997]	[0.9357, 0.9984]	[0.7844, 0.9936]	[0.4338, 0.9702]
clinc_dev@de	[0.8307, 0.9332]	[0.6936, 0.8781]	[0.9752, 0.9980]	[0.0520, 0.1201]	[0.9739, 0.9938]	[0.9699, 0.9974]	[0.8895, 0.9897]	[0.6289, 0.9527]
clinc_dev@lt	[0.8087, 0.9103]	[0.6500, 0.8336]	[0.9735, 0.9972]	[0.0749, 0.1465]	[0.9630, 0.9876]	[0.9673, 0.9961]	[0.8810, 0.9845]	[0.6092, 0.9303]
clinc_dev@pl	[0.8171, 0.9125]	[0.6628, 0.8404]	[0.9715, 0.9973]	[0.0683, 0.1400]	[0.9634, 0.9882]	[0.9646, 0.9964]	[0.8721, 0.9858]	[0.5894, 0.9361]
clinc_dev@sk	[0.8037, 0.9120]	[0.6361, 0.8356]	[0.9789, 0.9974]	[0.0686, 0.1451]	[0.9588, 0.9892]	[0.9730, 0.9964]	[0.9000, 0.9859]	[0.6546, 0.9363]
dstc11_banking_guard	[0.9045, 1.0000]	[0.9819, 1.0000]	[0.7864, 1.0000]	[0.0098, 0.0744]	[0.9752, 1.0000]	[0.8238, 1.0000]	[0.5390, 1.0000]	[0.1975, 1.0000]
dstc11_banking_guard@de	[0.8880, 1.0000]	[1.0000, 1.0000]	[0.7372, 1.0000]	[0.0109, 0.0857]	[0.9551, 1.0000]	[0.7919, 1.0000]	[0.4875, 1.0000]	[0.1669, 1.0000]
dstc11_banking_guard@lt	[0.8384, 0.9922]	[0.9310, 1.0000]	[0.6835, 1.0000]	[0.0192, 0.1046]	[0.9351, 1.0000]	[0.7499, 1.0000]	[0.4284, 1.0000]	[0.1363, 1.0000]
dstc11_banking_guard@pl	[0.8513, 1.0000]	[0.8485, 1.0000]	[0.7500, 1.0000]	[0.0141, 0.0916]	[0.9507, 1.0000]	[0.7926, 1.0000]	[0.4885, 1.0000]	[0.1674, 1.0000]
dstc11_banking_guard@sk	[0.8685, 0.9940]	[0.8889, 1.0000]	[0.7943, 1.0000]	[0.0150, 0.1017]	[0.9329, 1.0000]	[0.8178, 1.0000]	[0.5288, 1.0000]	[0.1911, 1.0000]
dstc11_finance_guard	[0.9256, 0.9790]	[0.9047, 0.9803]	[0.9381, 0.9864]	[0.0186, 0.0548]	[0.9794, 0.9969]	[0.9373, 0.9862]	[0.7890, 0.9471]	[0.4405, 0.7902]
dstc11_finance_guard@de	[0.9132, 0.9762]	[0.8696, 0.9756]	[0.9333, 0.9929]	[0.0246, 0.0707]	[0.9702, 0.9970]	[0.9331, 0.9922]	[0.7772, 0.9696]	[0.4234, 0.8702]
dstc11_finance_guard@lt	[0.9092, 0.9728]	[0.8615, 0.9740]	[0.9350, 0.9905]	[0.0290, 0.0767]	[0.9609, 0.9939]	[0.9348, 0.9898]	[0.7820, 0.9606]	[0.4302, 0.8368]
dstc11_finance_guard@pl	[0.8703, 0.9572]	[0.8000, 0.9391]	[0.9359, 0.9925]	[0.0389, 0.0993]	[0.9443, 0.9914]	[0.9304, 0.9913]	[0.7697, 0.9661]	[0.4131, 0.8571]
dstc11_finance_guard@sk	[0.8559, 0.9527]	[0.7712, 0.9302]	[0.9349, 0.9940]	[0.0390, 0.1048]	[0.9447, 0.9924]	[0.9280, 0.9931]	[0.7632, 0.9729]	[0.4043, 0.8831]
dstc11_insurance	[0.9004, 0.9834]	[0.9834, 1.0000]	[0.7899, 0.9667]	[0.0182, 0.0717]	[0.9842, 0.9993]	[0.8254, 0.9675]	[0.5416, 0.8814]	[0.1992, 0.6102]
hint3_dev	[0.5619, 0.9606]	[0.3529, 0.9402]	[0.9555, 1.0000]	[0.0267, 0.3600]	[0.7109, 0.9992]	[0.9297, 1.0000]	[0.7677, 1.0000]	[0.4103, 1.0000]
ind_dev_unseen_electricity_grid_operator	[0.8223, 0.9292]	[0.9193, 0.9928]	[0.6581, 0.8733]	[0.0576, 0.1304]	[0.9441, 0.9865]	[0.7398, 0.8837]	[0.4155, 0.6551]	[0.1302, 0.2856]
ind_dev_unseen_electricity_grid_operator@lt	[0.7878, 0.9031]	[0.8658, 0.9907]	[0.6129, 0.8466]	[0.0785, 0.1485]	[0.8855, 0.9729]	[0.7104, 0.8539]	[0.3801, 0.5937]	[0.1143, 0.2353]
ind_dev_unseen_electricity_grid_operator@pl	[0.7613, 0.8796]	[0.7432, 0.9091]	[0.7017, 0.9104]	[0.0879, 0.1447]	[0.8781, 0.9573]	[0.7383, 0.9017]	[0.4136, 0.6963]	[0.1293, 0.3256]
ind_dev_unseen_property_management_company	[0.8780, 0.9412]	[0.9070, 0.9725]	[0.8236, 0.9273]	[0.0511, 0.0909]	[0.9526, 0.9849]	[0.8419, 0.9292]	[0.5710, 0.7665]	[0.2189, 0.4087]
ind_dev_unseen_property_management_company@lt	[0.8210, 0.9092]	[0.7895, 0.9206]	[0.79999, 0.9333]	[0.0748, 0.1240]	[0.9091, 0.9676]	[0.8112, 0.9278]	[0.5179, 0.7625]	[0.1845, 0.4033]
ind_dev_unseen_property_management_company@pl	[0.8428, 0.9208]	[0.7910, 0.9077]	[0.8626, 0.9621]	[0.0646, 0.1096]	[0.9311, 0.9810]	[0.8606, 0.9579]	[0.6069, 0.8506]	[0.2453, 0.5452]
ind_dev_unseen_tax_and_social_security_office	[0.8516, 0.9209]	[0.9358, 0.9912]	[0.7364, 0.8561]	[0.0564, 0.1013]	[0.9633, 0.9888]	[0.7866, 0.8708]	[0.4796, 0.6275]	[0.1625, 0.2618]
ind_dev_unseen_tax_and_social_security_office@lt	[0.8490, 0.9238]	[0.9231, 0.9810]	[0.7548, 0.8664]	[0.0608, 0.1061]	[0.9427, 0.9793]	[0.7937, 0.8785]	[0.4903, 0.6438]	[0.1684, 0.2756]
ind_dev_unseen_tax_and_social_security_office@pl	[0.8578, 0.9256]	[0.9319, 0.9901]	[0.7472, 0.8713]	[0.0596, 0.1000]	[0.9545, 0.9836]	[0.7923, 0.8818]	[0.4881, 0.6509]	[0.1672, 0.2819]
ind_dev_unseen_telecom_operator	[0.8483, 0.9430]	[0.7956, 0.9551]	[0.8443, 0.9687]	[0.0486, 0.1033]	[0.9343, 0.9880]	[0.8510, 0.9648]	[0.5882, 0.8725]	[0.2312, 0.5903]
ind_dev_unseen_telecom_operator@lt	[0.8117, 0.9203]	[0.7321, 0.9176]	[0.8390, 0.9660]	[0.0643, 0.1385]	[0.8989, 0.9774]	[0.8400, 0.9601]	[0.5675, 0.8574]	[0.2165, 0.5586]
ind_dev_unseen_telecom_operator@pl	[0.7725, 0.9217]	[0.6598, 0.9241]	[0.8278, 0.9662]	[0.0639, 0.1590]	[0.8753, 0.9831]	[0.8278, 0.9604]	[0.5457, 0.8583]	[0.2019, 0.5605]
sgd_unseen	[0.8621, 1.0000]	[0.8621, 1.0000]	[0.9999999999999998, 1.0000000000000002]	[0.0026, 0.1185]	[0.9880, 1.0000000000000016]	[0.9999999999999998, 1.0000000000000002]	[0.9999999999999991, 1.0000000000000009]	[0.9999999999999954, 1.0000000000000047]
synth_dev_seen_domain	[0.9206, 0.9657]	[0.8665, 0.9479]	[0.9635, 0.9939]	[0.0273, 0.0564]	[0.9860, 0.9969]	[0.9619, 0.9933]	[0.8632, 0.9736]	[0.5705, 0.8859]
synth_dev_seen_domain@de	[0.9292, 0.9737]	[0.8814, 0.9583]	[0.9649, 0.9968]	[0.0283, 0.0590]	[0.9818, 0.9963]	[0.9630, 0.9965]	[0.8669, 0.9863]	[0.5783, 0.9379]
synth_dev_seen_domain@lt	[0.8943, 0.9548]	[0.8245, 0.9241]	[0.9617, 0.9946]	[0.0376, 0.0734]	[0.9747, 0.9945]	[0.9565, 0.9941]	[0.8461, 0.9766]	[0.5365, 0.8979]
synth_dev_seen_domain@pl	[0.9090, 0.9617]	[0.8504, 0.9405]	[0.9583, 0.9945]	[0.0399, 0.0736]	[0.9719, 0.9925]	[0.9548, 0.9940]	[0.8409, 0.9764]	[0.5266, 0.8972]
synth_dev_seen_domain@sk	[0.8946, 0.9538]	[0.8322, 0.9280]	[0.9480, 0.9897]	[0.0378, 0.0729]	[0.9749, 0.9948]	[0.9432, 0.9888]	[0.8060, 0.9568]	[0.4665, 0.8235]
synth_dev_unseen_bookstore	[0.4179, 1.0000]	[1.0000, 1.0000]	[0.1875, 0.1875]	[0.0025, 0.3904]	[0.8638, 0.9045]	[0.5517, 0.5517]	[0.2353, 0.2353]	[0.0608, 0.0608]
synth_dev_unseen_bookstore@de	[0.5213, 1.0000]	[0.8333, 1.0000]	[0.3636, 0.3636]	[0.0022, 0.3737]	[0.8775, 0.9437]	[0.5774, 0.5995]	[0.2546, 0.2723]	[0.0671, 0.0730]
synth_dev_unseen_bookstore@lt	[0.4153, 1.0000]	[0.8333, 1.0000]	[0.2143, 0.2143]	[0.0058, 0.4194]	[0.7885, 0.8119]	[0.5296, 0.5516]	[0.2196, 0.2352]	[0.0559, 0.0608]
synth_dev_unseen_bookstore@pl	[0.5269, 1.0000]	[0.8333, 1.0000]	[0.3636, 0.3636]	[0.0313, 0.3734]	[0.8300, 0.8571]	[0.5774, 0.5995]	[0.2546, 0.2723]	[0.0671, 0.0730]
synth_dev_unseen_bookstore@sk	[0.4412, 1.0000]	[0.8333, 1.0000]	[0.2500, 0.2500]	[0.0022, 0.4354]	[0.8542, 0.8958]	[0.5385, 0.5610]	[0.2258, 0.2421]	[0.0579, 0.0630]
synth_dev_unseen_camera_store	[0.6405, 1.0000]	[0.7931, 1.0000]	[0.3529, 0.8706]	[0.0322, 0.2475]	[0.6796, 1.0000]	[0.5577, 0.8793]	[0.2397, 0.6456]	[0.0622, 0.2772]
synth_dev_unseen_car_dealership_service_department	[0.7756, 0.9636]	[0.7991, 0.9755]	[0.5159, 1.0000]	[0.0330, 0.1427]	[0.8790, 0.9962]	[0.6564, 1.0000]	[0.3233, 1.0000]	[0.0914, 1.0000]
synth_dev_unseen_car_wash_subscription	[0.7655, 0.9095]	[0.7826, 0.9706]	[0.5789, 0.9134]	[0.0666, 0.1443]	[0.8748, 0.9762]	[0.6876, 0.9110]	[0.3549, 0.7190]	[0.1038, 0.3501]
synth_dev_unseen_dental_practice	[0.8185, 0.9327]	[0.8594, 0.9769]	[0.7359, 0.8996]	[0.0550, 0.1159]	[0.9224, 0.9829]	[0.7732, 0.9044]	[0.4602, 0.7028]	[0.1522, 0.3323]
synth_dev_unseen_dental_practice@de	[0.8043, 0.9396]	[0.8218, 0.9839]	[0.7249, 0.9129]	[0.0564, 0.1256]	[0.9076, 0.9868]	[0.7662, 0.9159]	[0.4503, 0.7313]	[0.1471, 0.3643]
synth_dev_unseen_dental_practice@lt	[0.7453, 0.8947]	[0.6975, 0.9167]	[0.7633, 0.9034]	[0.0733, 0.1586]	[0.8582, 0.9700]	[0.7659, 0.8992]	[0.4499, 0.6903]	[0.1469, 0.3194]
synth_dev_unseen_dental_practice@pl	[0.7414, 0.8913]	[0.7049, 0.9111]	[0.7440, 0.9101]	[0.0788, 0.1496]	[0.8682, 0.9641]	[0.7497, 0.9032]	[0.4281, 0.6998]	[0.1362, 0.3292]
synth_dev_unseen_dental_practice@sk	[0.7575, 0.9127]	[0.7340, 0.9518]	[0.7375, 0.8994]	[0.0721, 0.1536]	[0.8663, 0.9700]	[0.7556, 0.8989]	[0.4359, 0.6898]	[0.1399, 0.3188]
synth_dev_unseen_ebook_platform	[0.8008, 0.9461]	[0.9597, 1.0000]	[0.5866, 0.8919]	[0.0451, 0.1534]	[0.9353, 0.9911]	[0.7059, 0.9004]	[0.3751, 0.6932]	[0.1122, 0.3224]
synth_dev_unseen_ebook_platform@de	[0.8474, 0.9528]	[0.9111, 1.0000]	[0.7126, 0.9248]	[0.0415, 0.1061]	[0.9540, 0.9940]	[0.7688, 0.9263]	[0.4540, 0.7587]	[0.1490, 0.3982]
synth_dev_unseen_ebook_platform@lt	[0.8117, 0.9309]	[0.9062, 0.9904]	[0.6392, 0.9065]	[0.0567, 0.1378]	[0.9293, 0.9835]	[0.7279, 0.9091]	[0.4007, 0.7143]	[0.1234, 0.3448]
synth_dev_unseen_ebook_platform@pl	[0.8107, 0.9145]	[0.8165, 0.9565]	[0.7111, 0.9252]	[0.0647, 0.1290]	[0.9098, 0.9767]	[0.7588, 0.9234]	[0.4402, 0.7508]	[0.1420, 0.3881]
synth_dev_unseen_ebook_platform@sk	[0.8555, 0.9535]	[0.9082, 1.0000]	[0.7316, 0.9323]	[0.0470, 0.1100]	[0.9441, 0.9874]	[0.7824, 0.9333]	[0.4734, 0.7778]	[0.1591, 0.4242]
synth_dev_unseen_home_appliance_repair	[0.8811, 0.9550]	[0.8649, 0.9765]	[0.8465, 0.9628]	[0.0387, 0.0827]	[0.9568, 0.9945]	[0.8571, 0.9607]	[0.6000, 0.8595]	[0.2400, 0.5629]
synth_dev_unseen_home_appliance_repair@de	[0.8468, 0.9501]	[0.8276, 0.9831]	[0.7777, 0.9726]	[0.0516, 0.1054]	[0.9373, 0.9862]	[0.8070, 0.9694]	[0.5111, 0.8880]	[0.1804, 0.6253]
synth_dev_unseen_home_appliance_repair@lt	[0.7567, 0.8852]	[0.7188, 0.8824]	[0.7470, 0.9513]	[0.0918, 0.1542]	[0.8641, 0.9530]	[0.7529, 0.9434]	[0.4323, 0.8065]	[0.1382, 0.4673]
synth_dev_unseen_home_appliance_repair@pl	[0.8148, 0.9492]	[0.7895, 0.9768]	[0.7265, 0.9867]	[0.0656, 0.1279]	[0.9148, 0.9790]	[0.7705, 0.9850]	[0.4563, 0.9425]	[0.1501, 0.7753]
synth_dev_unseen_home_appliance_repair@sk	[0.8076, 0.9032]	[0.7600, 0.9355]	[0.7759, 0.9747]	[0.0763, 0.1376]	[0.8860, 0.9665]	[0.7947, 0.9695]	[0.4918, 0.8882]	[0.1692, 0.6258]
synth_dev_unseen_online_auction_site	[0.7837, 0.9396]	[0.8621, 0.9915]	[0.5833, 0.9185]	[0.0486, 0.1504]	[0.9103, 0.9896]	[0.6947, 0.9190]	[0.3626, 0.7392]	[0.1069, 0.3737]
synth_dev_unseen_online_auction_site@de	[0.8640, 0.9552]	[0.9231, 1.0000]	[0.7335, 0.9343]	[0.0434, 0.1035]	[0.9368, 0.9918]	[0.7852, 0.9352]	[0.4774, 0.7829]	[0.1613, 0.4316]
synth_dev_unseen_online_auction_site@lt	[0.7906, 0.9220]	[0.8027, 0.9559]	[0.6834, 0.9399]	[0.0587, 0.1368]	[0.9038, 0.9826]	[0.7384, 0.9364]	[0.4138, 0.7863]	[0.1294, 0.4366]
synth_dev_unseen_online_auction_site@pl	[0.7707, 0.9101]	[0.7628, 0.9653]	[0.6556, 0.9050]	[0.0695, 0.1429]	[0.8857, 0.9721]	[0.7239, 0.8989]	[0.3959, 0.6897]	[0.1212, 0.3187]
synth_dev_unseen_online_auction_site@sk	[0.8101, 0.9249]	[0.8182, 0.9669]	[0.6970, 0.9353]	[0.0678, 0.1348]	[0.8983, 0.9743]	[0.7545, 0.9320]	[0.4344, 0.7740]	[0.1392, 0.4189]
synth_dev_unseen_student_services_office	[0.8796, 0.9512]	[0.8895, 0.9781]	[0.8158, 0.9513]	[0.0353, 0.0799]	[0.9670, 0.9934]	[0.8358, 0.9502]	[0.5600, 0.8266]	[0.2113, 0.5008]
synth_dev_unseen_tattoo_studio	[0.6396, 0.7907]	[0.7024, 0.9500]	[0.3869, 0.7627]	[0.1313, 0.2423]	[0.7549, 0.9002]	[0.5955, 0.7567]	[0.2691, 0.4374]	[0.0719, 0.1407]
synth_dev_unseen_tattoo_studio@de	[0.6553, 0.7938]	[0.6447, 0.8810]	[0.5052, 0.8400]	[0.1454, 0.2196]	[0.7643, 0.8742]	[0.6177, 0.8140]	[0.2877, 0.5225]	[0.0784, 0.1873]
synth_dev_unseen_tattoo_studio@lt	[0.6457, 0.7740]	[0.6250, 0.8715]	[0.4840, 0.8282]	[0.1569, 0.2304]	[0.7415, 0.8562]	[0.6100, 0.7935]	[0.2811, 0.4899]	[0.0760, 0.1682]
synth_dev_unseen_tattoo_studio@pl	[0.6436, 0.7522]	[0.5882, 0.8588]	[0.4935, 0.8122]	[0.1575, 0.2353]	[0.7337, 0.8555]	[0.6156, 0.7694]	[0.2859, 0.4548]	[0.0777, 0.1494]
synth_dev_unseen_tattoo_studio@sk	[0.6303, 0.7491]	[0.5405, 0.8033]	[0.5688, 0.8577]	[0.1669, 0.2372]	[0.7290, 0.8551]	[0.6286, 0.8063]	[0.2973, 0.5099]	[0.0818, 0.1797]
synth_dev_unseen_vpn_service	[0.7380, 0.9709]	[0.8039, 1.0000]	[0.5000, 1.0000]	[0.0430, 0.1677]	[0.8532, 0.9899]	[0.6399, 1.0000]	[0.3076, 1.0000]	[0.0855, 1.0000]
synth_dev_unseen_vpn_service@de	[0.6239, 0.9714]	[0.5484, 1.0000]	[0.7000, 1.0000]	[0.0317, 0.2368]	[0.8008, 1.0000]	[0.7024, 1.0000]	[0.3711, 1.0000]	[0.1105, 1.0000]
synth_dev_unseen_vpn_service@lt	[0.6011, 0.9000]	[0.4667, 0.9048]	[0.7951, 1.0000]	[0.0741, 0.2319]	[0.8781, 0.9868]	[0.7788, 1.0000]	[0.4681, 1.0000]	[0.1563, 1.0000]
synth_dev_unseen_vpn_service@pl	[0.6602, 0.8908]	[0.5610, 0.9147]	[0.7678, 1.0000]	[0.0623, 0.1944]	[0.8739, 0.9783]	[0.7605, 1.0000]	[0.4425, 1.0000]	[0.1432, 1.0000]
synth_dev_unseen_vpn_service@sk	[0.6291, 0.9774]	[0.5143, 1.0000]	[0.7778, 1.0000]	[0.0496, 0.2570]	[0.7767, 0.9967]	[0.7521, 1.0000]	[0.4314, 1.0000]	[0.1377, 1.0000]
synth_dev_unseen_waste_collection_service	[0.6727, 0.8364]	[0.8314, 0.9810]	[0.4281, 0.6962]	[0.1204, 0.2277]	[0.7988, 0.9423]	[0.6146, 0.7511]	[0.2851, 0.4301]	[0.0774, 0.1371]
synth_dev_unseen_yoga_studio	[0.7445, 0.8793]	[0.7886, 0.9358]	[0.5831, 0.8778]	[0.0972, 0.1746]	[0.8525, 0.9385]	[0.6856, 0.8741]	[0.3528, 0.6344]	[0.1029, 0.2676]
synth_dev_unseen_yoga_studio@de	[0.7516, 0.8784]	[0.7764, 0.9224]	[0.6202, 0.8969]	[0.0994, 0.1719]	[0.8331, 0.9337]	[0.7001, 0.8879]	[0.3685, 0.6644]	[0.1094, 0.2942]
synth_dev_unseen_yoga_studio@lt	[0.7054, 0.8336]	[0.7073, 0.8444]	[0.5958, 0.8954]	[0.1216, 0.1982]	[0.7970, 0.9070]	[0.6660, 0.8779]	[0.3327, 0.6424]	[0.0950, 0.2744]
synth_dev_unseen_yoga_studio@pl	[0.7113, 0.8696]	[0.7241, 0.9221]	[0.5947, 0.8822]	[0.1017, 0.2035]	[0.7886, 0.9394]	[0.6744, 0.8748]	[0.3411, 0.6359]	[0.0983, 0.2688]
synth_dev_unseen_yoga_studio@sk	[0.7088, 0.8377]	[0.6228, 0.8235]	[0.7326, 0.9237]	[0.1155, 0.1962]	[0.8092, 0.9209]	[0.7375, 0.9052]	[0.4126, 0.7048]	[0.1288, 0.3345]

information.by_pool	check.interval_resamples_not_finite.tnr	check.interval_resamples_not_finite.auc	check.interval_resamples_not_finite.right_yes_0.5	check.interval_resamples_not_finite.right_yes_0.2	check.interval_resamples_not_finite.right_yes_0.05	scores.interval_resamples_not_finite.ece	scores.interval_resamples_not_finite.cand_ece	scores.interval_resamples_not_finite.rem_ece_6 to 20
abcd_dev
abcd_dev@de
abcd_dev@lt
abcd_dev@pl
abcd_dev@sk
bitext_dev
clinc_dev@de
clinc_dev@lt
clinc_dev@pl
clinc_dev@sk
dstc11_banking_guard	1	1	1	1	1
dstc11_banking_guard@de	1	1	1	1	1
dstc11_banking_guard@lt	1	1	1	1	1
dstc11_banking_guard@pl	1	1	1	1	1
dstc11_banking_guard@sk	1	1	1	1	1
dstc11_finance_guard
dstc11_finance_guard@de
dstc11_finance_guard@lt
dstc11_finance_guard@pl
dstc11_finance_guard@sk
dstc11_insurance
hint3_dev	10	10	10	10	10	2	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_electricity_grid_operator@lt
ind_dev_unseen_electricity_grid_operator@pl
ind_dev_unseen_property_management_company
ind_dev_unseen_property_management_company@lt
ind_dev_unseen_property_management_company@pl
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_tax_and_social_security_office@lt
ind_dev_unseen_tax_and_social_security_office@pl
ind_dev_unseen_telecom_operator
ind_dev_unseen_telecom_operator@lt
ind_dev_unseen_telecom_operator@pl
sgd_unseen	290	290	290	290	290	42	42	42
synth_dev_seen_domain
synth_dev_seen_domain@de
synth_dev_seen_domain@lt
synth_dev_seen_domain@pl
synth_dev_seen_domain@sk
synth_dev_unseen_bookstore	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_bookstore@de	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_bookstore@lt	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_bookstore@pl	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_bookstore@sk	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_camera_store	125	125	125	125	125	9	9	9
synth_dev_unseen_car_dealership_service_department	62	62	62	62	62	5	5	5
synth_dev_unseen_car_wash_subscription	2	2	2	2	2
synth_dev_unseen_dental_practice
synth_dev_unseen_dental_practice@de
synth_dev_unseen_dental_practice@lt
synth_dev_unseen_dental_practice@pl
synth_dev_unseen_dental_practice@sk
synth_dev_unseen_ebook_platform
synth_dev_unseen_ebook_platform@de
synth_dev_unseen_ebook_platform@lt
synth_dev_unseen_ebook_platform@pl
synth_dev_unseen_ebook_platform@sk
synth_dev_unseen_home_appliance_repair	3	3	3	3	3	1	1	1
synth_dev_unseen_home_appliance_repair@de	3	3	3	3	3	1	1	1
synth_dev_unseen_home_appliance_repair@lt	3	3	3	3	3	1	1	1
synth_dev_unseen_home_appliance_repair@pl	3	3	3	3	3	1	1	1
synth_dev_unseen_home_appliance_repair@sk	3	3	3	3	3	1	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_online_auction_site@de
synth_dev_unseen_online_auction_site@lt
synth_dev_unseen_online_auction_site@pl
synth_dev_unseen_online_auction_site@sk
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_tattoo_studio@de
synth_dev_unseen_tattoo_studio@lt
synth_dev_unseen_tattoo_studio@pl
synth_dev_unseen_tattoo_studio@sk
synth_dev_unseen_vpn_service	22	22	22	22	22	1	1	1
synth_dev_unseen_vpn_service@de	22	22	22	22	22	1	1	1
synth_dev_unseen_vpn_service@lt	22	22	22	22	22	1	1	1
synth_dev_unseen_vpn_service@pl	22	22	22	22	22	1	1	1
synth_dev_unseen_vpn_service@sk	22	22	22	22	22	1	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio
synth_dev_unseen_yoga_studio@de
synth_dev_unseen_yoga_studio@lt
synth_dev_unseen_yoga_studio@pl
synth_dev_unseen_yoga_studio@sk

information.by_pool	scores.interval_resamples_not_finite.rem_ece	scores.interval_resamples_not_finite.recall_at_5	scores.interval_resamples_not_finite.accuracy	scores.interval_resamples_not_finite.mean_top	scores.interval_resamples_not_finite.mean_top_minus_accuracy	scores.interval_resamples_not_finite.brier	check.interval_resamples_not_finite.ece	check.interval_resamples_not_finite.ba
abcd_dev
abcd_dev@de
abcd_dev@lt
abcd_dev@pl
abcd_dev@sk
bitext_dev
clinc_dev@de
clinc_dev@lt
clinc_dev@pl
clinc_dev@sk
dstc11_banking_guard
dstc11_banking_guard@de
dstc11_banking_guard@lt
dstc11_banking_guard@pl
dstc11_banking_guard@sk
dstc11_finance_guard
dstc11_finance_guard@de
dstc11_finance_guard@lt
dstc11_finance_guard@pl
dstc11_finance_guard@sk
dstc11_insurance
hint3_dev	2	2	2	2	2	2	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_electricity_grid_operator@lt
ind_dev_unseen_electricity_grid_operator@pl
ind_dev_unseen_property_management_company
ind_dev_unseen_property_management_company@lt
ind_dev_unseen_property_management_company@pl
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_tax_and_social_security_office@lt
ind_dev_unseen_tax_and_social_security_office@pl
ind_dev_unseen_telecom_operator
ind_dev_unseen_telecom_operator@lt
ind_dev_unseen_telecom_operator@pl
sgd_unseen	42	42	42	42	42	42	42	42
synth_dev_seen_domain
synth_dev_seen_domain@de
synth_dev_seen_domain@lt
synth_dev_seen_domain@pl
synth_dev_seen_domain@sk
synth_dev_unseen_bookstore		290	290	290	290	290	290	290
synth_dev_unseen_bookstore@de		290	290	290	290	290	290	290
synth_dev_unseen_bookstore@lt		290	290	290	290	290	290	290
synth_dev_unseen_bookstore@pl		290	290	290	290	290	290	290
synth_dev_unseen_bookstore@sk		290	290	290	290	290	290	290
synth_dev_unseen_camera_store	9	9	9	9	9	9	9	9
synth_dev_unseen_car_dealership_service_department	5	5	5	5	5	5	5	5
synth_dev_unseen_car_wash_subscription
synth_dev_unseen_dental_practice
synth_dev_unseen_dental_practice@de
synth_dev_unseen_dental_practice@lt
synth_dev_unseen_dental_practice@pl
synth_dev_unseen_dental_practice@sk
synth_dev_unseen_ebook_platform
synth_dev_unseen_ebook_platform@de
synth_dev_unseen_ebook_platform@lt
synth_dev_unseen_ebook_platform@pl
synth_dev_unseen_ebook_platform@sk
synth_dev_unseen_home_appliance_repair	1	1	1	1	1	1	1	1
synth_dev_unseen_home_appliance_repair@de	1	1	1	1	1	1	1	1
synth_dev_unseen_home_appliance_repair@lt	1	1	1	1	1	1	1	1
synth_dev_unseen_home_appliance_repair@pl	1	1	1	1	1	1	1	1
synth_dev_unseen_home_appliance_repair@sk	1	1	1	1	1	1	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_online_auction_site@de
synth_dev_unseen_online_auction_site@lt
synth_dev_unseen_online_auction_site@pl
synth_dev_unseen_online_auction_site@sk
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_tattoo_studio@de
synth_dev_unseen_tattoo_studio@lt
synth_dev_unseen_tattoo_studio@pl
synth_dev_unseen_tattoo_studio@sk
synth_dev_unseen_vpn_service	1	1	1	1	1	1	1	1
synth_dev_unseen_vpn_service@de	1	1	1	1	1	1	1	1
synth_dev_unseen_vpn_service@lt	1	1	1	1	1	1	1	1
synth_dev_unseen_vpn_service@pl	1	1	1	1	1	1	1	1
synth_dev_unseen_vpn_service@sk	1	1	1	1	1	1	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio
synth_dev_unseen_yoga_studio@de
synth_dev_unseen_yoga_studio@lt
synth_dev_unseen_yoga_studio@pl
synth_dev_unseen_yoga_studio@sk

information.by_pool	check.interval_resamples_not_finite.tyr	check.interval_resamples_not_finite.brier
abcd_dev
abcd_dev@de
abcd_dev@lt
abcd_dev@pl
abcd_dev@sk
bitext_dev
clinc_dev@de
clinc_dev@lt
clinc_dev@pl
clinc_dev@sk
dstc11_banking_guard
dstc11_banking_guard@de
dstc11_banking_guard@lt
dstc11_banking_guard@pl
dstc11_banking_guard@sk
dstc11_finance_guard
dstc11_finance_guard@de
dstc11_finance_guard@lt
dstc11_finance_guard@pl
dstc11_finance_guard@sk
dstc11_insurance
hint3_dev	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_electricity_grid_operator@lt
ind_dev_unseen_electricity_grid_operator@pl
ind_dev_unseen_property_management_company
ind_dev_unseen_property_management_company@lt
ind_dev_unseen_property_management_company@pl
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_tax_and_social_security_office@lt
ind_dev_unseen_tax_and_social_security_office@pl
ind_dev_unseen_telecom_operator
ind_dev_unseen_telecom_operator@lt
ind_dev_unseen_telecom_operator@pl
sgd_unseen	42	42
synth_dev_seen_domain
synth_dev_seen_domain@de
synth_dev_seen_domain@lt
synth_dev_seen_domain@pl
synth_dev_seen_domain@sk
synth_dev_unseen_bookstore	290	290
synth_dev_unseen_bookstore@de	290	290
synth_dev_unseen_bookstore@lt	290	290
synth_dev_unseen_bookstore@pl	290	290
synth_dev_unseen_bookstore@sk	290	290
synth_dev_unseen_camera_store	9	9
synth_dev_unseen_car_dealership_service_department	5	5
synth_dev_unseen_car_wash_subscription
synth_dev_unseen_dental_practice
synth_dev_unseen_dental_practice@de
synth_dev_unseen_dental_practice@lt
synth_dev_unseen_dental_practice@pl
synth_dev_unseen_dental_practice@sk
synth_dev_unseen_ebook_platform
synth_dev_unseen_ebook_platform@de
synth_dev_unseen_ebook_platform@lt
synth_dev_unseen_ebook_platform@pl
synth_dev_unseen_ebook_platform@sk
synth_dev_unseen_home_appliance_repair	1	1
synth_dev_unseen_home_appliance_repair@de	1	1
synth_dev_unseen_home_appliance_repair@lt	1	1
synth_dev_unseen_home_appliance_repair@pl	1	1
synth_dev_unseen_home_appliance_repair@sk	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_online_auction_site@de
synth_dev_unseen_online_auction_site@lt
synth_dev_unseen_online_auction_site@pl
synth_dev_unseen_online_auction_site@sk
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_tattoo_studio@de
synth_dev_unseen_tattoo_studio@lt
synth_dev_unseen_tattoo_studio@pl
synth_dev_unseen_tattoo_studio@sk
synth_dev_unseen_vpn_service	1	1
synth_dev_unseen_vpn_service@de	1	1
synth_dev_unseen_vpn_service@lt	1	1
synth_dev_unseen_vpn_service@pl	1	1
synth_dev_unseen_vpn_service@sk	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio
synth_dev_unseen_yoga_studio@de
synth_dev_unseen_yoga_studio@lt
synth_dev_unseen_yoga_studio@pl
synth_dev_unseen_yoga_studio@sk

information.by_family	scores.rows	scores.ece	scores.ece_p975	scores.ece_interval	scores.cand_ece	scores.cand_ece_p975	scores.cand_ece_interval	scores.rem_ece
abcd_dev	1,525	0.0890	0.1702	[0.0503, 0.1702]	0.0279	0.0491	[0.0161, 0.0491]	0.0883
bitext_dev	508	0.0708	0.1584	[0.0359, 0.1584]	0.0203	0.0519	[0.0123, 0.0519]	0.0136
clinc_dev	1,911	0.0281	0.0790	[0.0216, 0.0790]	0.0097	0.0239	[0.0091, 0.0239]	0.0452
dstc11_banking_guard	396	0.0555	0.0902	[0.0363, 0.0902]	0.0187	0.0291	[0.0130, 0.0291]	0.0070
dstc11_finance_guard	1,121	0.0338	0.0940	[0.0264, 0.0940]	0.0104	0.0333	[0.0082, 0.0333]	0.0131
dstc11_insurance	377	0.0418	0.1420	[0.0137, 0.1420]	0.0176	0.0590	[0.0049, 0.0590]	0.0096
hint3_dev	67	0.1526	n/a	[0.0375, 0.3446]	0.0550	n/a	[0.0177, 0.1112]	0.1078
ind_dev_unseen_electricity_grid_operator	343	0.0927	0.1679	[0.0649, 0.1679]	0.0309	0.0525	[0.0208, 0.0525]	0.03002
ind_dev_unseen_property_management_company	642	0.0599	0.1080	[0.0527, 0.1080]	0.0181	0.0334	[0.0152, 0.0334]	0.0218
ind_dev_unseen_tax_and_social_security_office	635	0.1021	0.1489	[0.0710, 0.1489]	0.0304	0.0434	[0.0245, 0.0434]	0.0594
ind_dev_unseen_telecom_operator	386	0.0566	0.1630	[0.0422, 0.1630]	0.0179	0.0582	[0.0124, 0.0582]	0.0201
sgd_unseen	102	0.0987	n/a	[0.0046, 0.3476]	0.0397	n/a	[0.0018, 0.1382]	0.0010
synth_dev_seen_domain	1,571	0.0451	0.0754	[0.0331, 0.0754]	0.0145	0.0247	[0.0093, 0.0247]	0.0436
synth_dev_unseen_bookstore	64	0.0753	n/a	[0.0397, 0.1203]	0.0376	n/a	[0.0198, 0.0561]	n/a
synth_dev_unseen_camera_store	45	0.1703	n/a	[0.0970, 0.3688]	0.0574	n/a	[0.0336, 0.1018]	0.0088
synth_dev_unseen_car_dealership_service_department	57	0.1353	n/a	[0.0826, 0.2213]	0.0471	n/a	[0.0423, 0.0859]	0.0152
synth_dev_unseen_car_wash_subscription	73	0.1077	0.2724	[0.0887, 0.2724]	0.0377	0.0881	[0.0313, 0.0881]	0.0305
synth_dev_unseen_dental_practice	609	0.0503	0.1375	[0.0485, 0.1375]	0.0184	0.0476	[0.0175, 0.0476]	0.0326
synth_dev_unseen_ebook_platform	534	0.0408	0.1020	[0.0386, 0.1020]	0.0112	0.0315	[0.0114, 0.0315]	0.0170
synth_dev_unseen_home_appliance_repair	344	0.0683	n/a	[0.0642, 0.1157]	0.03003	n/a	[0.0263, 0.0430]	0.0099
synth_dev_unseen_online_auction_site	471	0.0716	0.1235	[0.0635, 0.1235]	0.0286	0.0456	[0.0228, 0.0456]	0.0143
synth_dev_unseen_student_services_office	177	0.0706	0.1365	[0.0565, 0.1365]	0.0222	0.0436	[0.0183, 0.0436]	0.0309
synth_dev_unseen_tattoo_studio	470	0.1093	0.2189	[0.0848, 0.2189]	0.0488	0.0782	[0.0385, 0.0782]	0.0733
synth_dev_unseen_vpn_service	264	0.1691	n/a	[0.1188, 0.2577]	0.0486	n/a	[0.0362, 0.0856]	0.0201
synth_dev_unseen_waste_collection_service	168	0.1023	0.1952	[0.0895, 0.1952]	0.0314	0.0633	[0.0291, 0.0633]	0.0703
synth_dev_unseen_yoga_studio	545	0.0799	0.1673	[0.0523, 0.1673]	0.0261	0.0542	[0.0229, 0.0542]	0.0606

information.by_family	scores.rem_ece_p975	scores.rem_ece_bands.6 to 20	scores.rem_ece_bands.21 to 40	scores.rem_ece_bands.41 or more	scores.remainder_info.recall_at_5	scores.remainder_info.over_5.rows	scores.remainder_info.over_5.not_read_share	scores.remainder_info.over_5.mean_remainder
abcd_dev	0.1723	n/a	n/a	0.0883	0.7639	1,525	0.2361	0.1490
bitext_dev	0.0232	n/a	0.0136	n/a	1.0000	508	0.0000	0.0136
clinc_dev	0.0760	n/a	n/a	0.0452	0.9529	1,911	0.0471	0.0923
dstc11_banking_guard	0.0093	0.0070	n/a	n/a	1.0000	396	0.0000	0.0070
dstc11_finance_guard	0.0332	n/a	0.0131	n/a	0.9759	1,121	0.0241	0.0331
dstc11_insurance	0.0160	n/a	0.0096	n/a	0.9973	377	0.0027	0.0099
hint3_dev	n/a	0.1078	n/a	n/a	0.8806	67	0.1194	0.0151
ind_dev_unseen_electricity_grid_operator	0.0538	n/a	0.03002	n/a	0.9417	343	0.0583	0.0722
ind_dev_unseen_property_management_company	0.0623	n/a	n/a	0.0218	0.9081	642	0.0919	0.0969
ind_dev_unseen_tax_and_social_security_office	0.0833	n/a	n/a	0.0594	0.9638	635	0.0362	0.0938
ind_dev_unseen_telecom_operator	0.0479	n/a	0.0201	n/a	0.9508	386	0.0492	0.0621
sgd_unseen	n/a	0.0010	n/a	n/a	1.0000	102	0.0000	0.0010
synth_dev_seen_domain	0.0680	n/a	n/a	0.0436	0.9453	1,571	0.0547	0.0984
synth_dev_unseen_bookstore	n/a	n/a	n/a	n/a	1.0000	0	n/a	n/a
synth_dev_unseen_camera_store	n/a	0.0088	n/a	n/a	1.0000	45	0.0000	0.0088
synth_dev_unseen_car_dealership_service_department	n/a	0.0152	n/a	n/a	1.0000	57	0.0000	0.0152
synth_dev_unseen_car_wash_subscription	0.1198	0.0305	n/a	n/a	0.9452	73	0.0548	0.0286
synth_dev_unseen_dental_practice	0.0776	n/a	0.0326	n/a	0.9294	609	0.0706	0.0770
synth_dev_unseen_ebook_platform	0.0437	0.0170	n/a	n/a	0.9775	534	0.0225	0.0395
synth_dev_unseen_home_appliance_repair	n/a	0.0099	n/a	n/a	0.9535	344	0.0465	0.0366
synth_dev_unseen_online_auction_site	0.0328	0.0143	n/a	n/a	0.9766	471	0.0234	0.0376
synth_dev_unseen_student_services_office	0.0507	n/a	0.0309	n/a	0.9774	177	0.0226	0.0374
synth_dev_unseen_tattoo_studio	0.2151	0.0733	n/a	n/a	0.8404	470	0.1596	0.0862
synth_dev_unseen_vpn_service	n/a	0.0201	n/a	n/a	0.9962	264	0.0038	0.0238
synth_dev_unseen_waste_collection_service	0.1455	n/a	0.0703	n/a	0.8631	168	0.1369	0.1095
synth_dev_unseen_yoga_studio	0.1399	n/a	0.0606	n/a	0.8532	545	0.1468	0.0862

information.by_family	scores.remainder_info.over_5.remainder_ece	scores.remainder_info.by_size.6 to 20.rows	scores.remainder_info.by_size.6 to 20.not_read_share	scores.remainder_info.by_size.6 to 20.mean_remainder	scores.remainder_info.by_size.6 to 20.remainder_ece	scores.remainder_info.by_size.21 to 40.rows	scores.remainder_info.by_size.21 to 40.not_read_share	scores.remainder_info.by_size.21 to 40.mean_remainder
abcd_dev	0.0883	0	n/a	n/a	n/a	0	n/a	n/a
bitext_dev	0.0136	0	n/a	n/a	n/a	508	0.0000	0.0136
clinc_dev	0.0452	0	n/a	n/a	n/a	0	n/a	n/a
dstc11_banking_guard	0.0070	396	0.0000	0.0070	0.0070	0	n/a	n/a
dstc11_finance_guard	0.0131	0	n/a	n/a	n/a	1,121	0.0241	0.0331
dstc11_insurance	0.0096	0	n/a	n/a	n/a	377	0.0027	0.0099
hint3_dev	0.1078	67	0.1194	0.0151	0.1078	0	n/a	n/a
ind_dev_unseen_electricity_grid_operator	0.03002	0	n/a	n/a	n/a	343	0.0583	0.0722
ind_dev_unseen_property_management_company	0.0218	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_tax_and_social_security_office	0.0594	0	n/a	n/a	n/a	0	n/a	n/a
ind_dev_unseen_telecom_operator	0.0201	0	n/a	n/a	n/a	386	0.0492	0.0621
sgd_unseen	0.0010	102	0.0000	0.0010	0.0010	0	n/a	n/a
synth_dev_seen_domain	0.0436	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_bookstore	n/a	0	n/a	n/a	n/a	0	n/a	n/a
synth_dev_unseen_camera_store	0.0088	45	0.0000	0.0088	0.0088	0	n/a	n/a
synth_dev_unseen_car_dealership_service_department	0.0152	57	0.0000	0.0152	0.0152	0	n/a	n/a
synth_dev_unseen_car_wash_subscription	0.0305	73	0.0548	0.0286	0.0305	0	n/a	n/a
synth_dev_unseen_dental_practice	0.0326	0	n/a	n/a	n/a	609	0.0706	0.0770
synth_dev_unseen_ebook_platform	0.0170	534	0.0225	0.0395	0.0170	0	n/a	n/a
synth_dev_unseen_home_appliance_repair	0.0099	344	0.0465	0.0366	0.0099	0	n/a	n/a
synth_dev_unseen_online_auction_site	0.0143	471	0.0234	0.0376	0.0143	0	n/a	n/a
synth_dev_unseen_student_services_office	0.0309	0	n/a	n/a	n/a	177	0.0226	0.0374
synth_dev_unseen_tattoo_studio	0.0733	470	0.1596	0.0862	0.0733	0	n/a	n/a
synth_dev_unseen_vpn_service	0.0201	264	0.0038	0.0238	0.0201	0	n/a	n/a
synth_dev_unseen_waste_collection_service	0.0703	0	n/a	n/a	n/a	168	0.1369	0.1095
synth_dev_unseen_yoga_studio	0.0606	0	n/a	n/a	n/a	545	0.1468	0.0862

information.by_family	scores.remainder_info.by_size.21 to 40.remainder_ece	scores.remainder_info.by_size.41 or more.rows	scores.remainder_info.by_size.41 or more.not_read_share	scores.remainder_info.by_size.41 or more.mean_remainder	scores.remainder_info.by_size.41 or more.remainder_ece	scores.rem_ece_band_intervals.6 to 20	scores.rem_ece_band_intervals.21 to 40	scores.rem_ece_band_intervals.41 or more
abcd_dev	n/a	1,525	0.2361	0.1490	0.0883	n/a	n/a	[0.0385, 0.1723]
bitext_dev	0.0136	0	n/a	n/a	n/a	n/a	[0.0051, 0.0232]	n/a
clinc_dev	n/a	1,911	0.0471	0.0923	0.0452	n/a	n/a	[0.0257, 0.0760]
dstc11_banking_guard	n/a	0	n/a	n/a	n/a	[0.0048, 0.0093]	n/a	n/a
dstc11_finance_guard	0.0131	0	n/a	n/a	n/a	n/a	[0.0065, 0.0332]	n/a
dstc11_insurance	0.0096	0	n/a	n/a	n/a	n/a	[0.0063, 0.0160]	n/a
hint3_dev	n/a	0	n/a	n/a	n/a	[0.0038, 0.2910]	n/a	n/a
ind_dev_unseen_electricity_grid_operator	0.03002	0	n/a	n/a	n/a	n/a	[0.0156, 0.0538]	n/a
ind_dev_unseen_property_management_company	n/a	642	0.0919	0.0969	0.0218	n/a	n/a	[0.0208, 0.0623]
ind_dev_unseen_tax_and_social_security_office	n/a	635	0.0362	0.0938	0.0594	n/a	n/a	[0.0430, 0.0833]
ind_dev_unseen_telecom_operator	0.0201	0	n/a	n/a	n/a	n/a	[0.0139, 0.0479]	n/a
sgd_unseen	n/a	0	n/a	n/a	n/a	[0.0002, 0.0027]	n/a	n/a
synth_dev_seen_domain	n/a	1,571	0.0547	0.0984	0.0436	n/a	n/a	[0.0304, 0.0680]
synth_dev_unseen_bookstore	n/a	0	n/a	n/a	n/a	n/a	n/a	n/a
synth_dev_unseen_camera_store	n/a	0	n/a	n/a	n/a	[0.0028, 0.0140]	n/a	n/a
synth_dev_unseen_car_dealership_service_department	n/a	0	n/a	n/a	n/a	[0.0088, 0.0248]	n/a	n/a
synth_dev_unseen_car_wash_subscription	n/a	0	n/a	n/a	n/a	[0.0168, 0.1198]	n/a	n/a
synth_dev_unseen_dental_practice	0.0326	0	n/a	n/a	n/a	n/a	[0.0212, 0.0776]	n/a
synth_dev_unseen_ebook_platform	n/a	0	n/a	n/a	n/a	[0.0075, 0.0437]	n/a	n/a
synth_dev_unseen_home_appliance_repair	n/a	0	n/a	n/a	n/a	[0.0048, 0.0641]	n/a	n/a
synth_dev_unseen_online_auction_site	n/a	0	n/a	n/a	n/a	[0.0099, 0.0328]	n/a	n/a
synth_dev_unseen_student_services_office	0.0309	0	n/a	n/a	n/a	n/a	[0.0216, 0.0507]	n/a
synth_dev_unseen_tattoo_studio	n/a	0	n/a	n/a	n/a	[0.0228, 0.2151]	n/a	n/a
synth_dev_unseen_vpn_service	n/a	0	n/a	n/a	n/a	[0.0103, 0.0366]	n/a	n/a
synth_dev_unseen_waste_collection_service	0.0703	0	n/a	n/a	n/a	n/a	[0.0367, 0.1455]	n/a
synth_dev_unseen_yoga_studio	0.0606	0	n/a	n/a	n/a	n/a	[0.0191, 0.1399]	n/a

information.by_family	scores.rem_ece_interval	scores.recall_at_5	scores.recall_at_5_interval	scores.accuracy	scores.accuracy_interval	scores.mean_top	scores.mean_top_interval	scores.mean_top_minus_accuracy
abcd_dev	[0.0385, 0.1723]	0.7639	[0.6782, 0.8393]	0.5357	[0.4387, 0.6248]	0.6245	[0.5799, 0.6676]	0.0887
bitext_dev	[0.0051, 0.0232]	1.0000	[1.0000, 1.0000]	0.9232	[0.8504, 0.9861]	0.8846	[0.8224, 0.9432]	-0.0386
clinc_dev	[0.0257, 0.0760]	0.9529	[0.9127, 0.9827]	0.7708	[0.6890, 0.8441]	0.7460	[0.6905, 0.7982]	-0.0248
dstc11_banking_guard	[0.0048, 0.0093]	1.0000	[1.0000, 1.0000]	0.9798	[0.9528, 0.9970]	0.9245	[0.8858, 0.9607]	-0.0553
dstc11_finance_guard	[0.0065, 0.0332]	0.9759	[0.9561, 0.9924]	0.8537	[0.7701, 0.9260]	0.8696	[0.8193, 0.9170]	0.0159
dstc11_insurance	[0.0063, 0.0160]	0.9973	[0.9911, 1.0000]	0.9178	[0.7838, 0.9858]	0.9508	[0.9092, 0.9774]	0.0330
hint3_dev	[0.0038, 0.2910]	0.8806	[0.6920, 1.0000]	0.7463	[0.4938, 0.9592]	0.8759	[0.8023, 0.9753]	0.1296
ind_dev_unseen_electricity_grid_operator	[0.0156, 0.0538]	0.9417	[0.9134, 0.9684]	0.7784	[0.6934, 0.8603]	0.6953	[0.6496, 0.7423]	-0.0831
ind_dev_unseen_property_management_company	[0.0208, 0.0623]	0.9081	[0.8542, 0.9538]	0.7134	[0.6177, 0.8019]	0.7144	[0.6753, 0.7547]	0.0010
ind_dev_unseen_tax_and_social_security_office	[0.0430, 0.0833]	0.9638	[0.9356, 0.9856]	0.8157	[0.7543, 0.8743]	0.7191	[0.6750, 0.7604]	-0.0967
ind_dev_unseen_telecom_operator	[0.0139, 0.0479]	0.9508	[0.9052, 0.9855]	0.7047	[0.5613, 0.8229]	0.7349	[0.6770, 0.7941]	0.0303
sgd_unseen	[0.0002, 0.0027]	1.0000	[1.0000, 1.0000]	0.8529	[0.5333, 1.0000]	0.9394	[0.8556, 0.9954]	0.0865
synth_dev_seen_domain	[0.0304, 0.0680]	0.9453	[0.9190, 0.9705]	0.8351	[0.7761, 0.8910]	0.7925	[0.7420, 0.8427]	-0.0426
synth_dev_unseen_bookstore	n/a	1.0000	[1.0000, 1.0000]	0.9688	[0.9429, 1.0000]	0.9291	[0.9031, 0.9603]	-0.0397
synth_dev_unseen_camera_store	[0.0028, 0.0140]	1.0000	[1.0000, 1.0000]	0.8667	[0.8000, 0.9474]	0.7669	[0.6401, 0.8994]	-0.0998
synth_dev_unseen_car_dealership_service_department	[0.0088, 0.0248]	1.0000	[1.0000, 1.0000]	0.8947	[0.8333, 0.9474]	0.7933	[0.7361, 0.8497]	-0.1015
synth_dev_unseen_car_wash_subscription	[0.0168, 0.1198]	0.9452	[0.8548, 1.0000]	0.7123	[0.5652, 0.8594]	0.7666	[0.7181, 0.8097]	0.0543
synth_dev_unseen_dental_practice	[0.0212, 0.0776]	0.9294	[0.8564, 0.9849]	0.7011	[0.5827, 0.8115]	0.6605	[0.6137, 0.7060]	-0.0406
synth_dev_unseen_ebook_platform	[0.0075, 0.0437]	0.9775	[0.9486, 0.9965]	0.8221	[0.7198, 0.9028]	0.7931	[0.7301, 0.8516]	-0.0290
synth_dev_unseen_home_appliance_repair	[0.0048, 0.0641]	0.9535	[0.9011, 0.9957]	0.7878	[0.7074, 0.8707]	0.7598	[0.7024, 0.8167]	-0.0280
synth_dev_unseen_online_auction_site	[0.0099, 0.0328]	0.9766	[0.9583, 0.9940]	0.8556	[0.7824, 0.9270]	0.7870	[0.7355, 0.8396]	-0.0686
synth_dev_unseen_student_services_office	[0.0216, 0.0507]	0.9774	[0.9485, 1.0000]	0.8927	[0.8143, 0.9600]	0.8559	[0.8010, 0.9069]	-0.0367
synth_dev_unseen_tattoo_studio	[0.0228, 0.2151]	0.8404	[0.7002, 0.9495]	0.6468	[0.4542, 0.8138]	0.5735	[0.4958, 0.6561]	-0.0733
synth_dev_unseen_vpn_service	[0.0103, 0.0366]	0.9962	[0.9865, 1.0000]	0.9053	[0.8056, 0.9818]	0.8065	[0.7219, 0.8914]	-0.0988
synth_dev_unseen_waste_collection_service	[0.0367, 0.1455]	0.8631	[0.7558, 0.9551]	0.6071	[0.4540, 0.7695]	0.6045	[0.5274, 0.6903]	-0.0026
synth_dev_unseen_yoga_studio	[0.0191, 0.1399]	0.8532	[0.7644, 0.9237]	0.5817	[0.4794, 0.6900]	0.6081	[0.5565, 0.6554]	0.0264

information.by_family	scores.mean_top_minus_accuracy_interval	scores.brier	scores.brier_interval	check.pairs	check.ece	check.ba	check.tyr	check.tnr
abcd_dev	[0.0196, 0.1661]	0.5794	[0.4702, 0.6912]	7,625	0.0951	0.8008	0.6852	0.9164
bitext_dev	[-0.0970, 0.0138]	0.1362	[0.0417, 0.2614]	2,540	0.0510	0.9473	0.9213	0.9734
clinc_dev	[-0.0732, 0.0301]	0.3145	[0.2215, 0.4132]	9,555	0.1284	0.8736	0.7582	0.9889
dstc11_banking_guard	[-0.0791, -0.0304]	0.0470	[0.0168, 0.0890]	1,980	0.0487	0.9495	0.9823	0.9167
dstc11_finance_guard	[-0.0347, 0.0769]	0.2160	[0.1142, 0.3401]	5,605	0.0372	0.9428	0.9179	0.9677
dstc11_insurance	[-0.0197, 0.1270]	0.1144	[0.0269, 0.2798]	1,885	0.0834	0.9456	0.9920	0.8992
hint3_dev	[-0.0361, 0.3446]	0.4070	[0.0768, 0.7819]	335	0.1752	0.8004	0.6119	0.9888
ind_dev_unseen_electricity_grid_operator	[-0.1481, -0.0218]	0.3369	[0.2423, 0.4335]	1,715	0.0808	0.8495	0.9184	0.7806
ind_dev_unseen_property_management_company	[-0.0609, 0.0704]	0.3626	[0.2758, 0.4614]	3,210	0.0357	0.8910	0.8910	0.8910
ind_dev_unseen_tax_and_social_security_office	[-0.1450, -0.0500]	0.2786	[0.2141, 0.3480]	3,175	0.0860	0.8864	0.9622	0.8106
ind_dev_unseen_telecom_operator	[-0.0638, 0.1448]	0.4088	[0.2562, 0.5839]	1,930	0.0446	0.8799	0.8446	0.9152
sgd_unseen	[-0.0198, 0.3223]	0.2496	[0.0006, 0.8005]	408	0.0632	0.9559	0.9118	1.0000
synth_dev_seen_domain	[-0.0720, -0.0122]	0.2243	[0.1585, 0.2913]	7,855	0.0577	0.9390	0.8982	0.9798
synth_dev_unseen_bookstore	[-0.0397, -0.0397]	0.0495	[0.0081, 0.0839]	128	0.3220	0.6016	0.9375	0.2656
synth_dev_unseen_camera_store	[-0.1599, -0.0420]	0.2657	[0.0956, 0.4133]	225	0.1308	0.7472	0.9111	0.5833
synth_dev_unseen_car_dealership_service_department	[-0.1896, 0.0029]	0.1770	[0.1359, 0.2242]	285	0.0913	0.8618	0.8947	0.8289
synth_dev_unseen_car_wash_subscription	[-0.0655, 0.1935]	0.4031	[0.2069, 0.6202]	365	0.0814	0.8288	0.8904	0.7671
synth_dev_unseen_dental_practice	[-0.1244, 0.0458]	0.4257	[0.3014, 0.5606]	3,045	0.0450	0.84996	0.8670	0.8329
synth_dev_unseen_ebook_platform	[-0.0895, 0.0402]	0.2525	[0.1475, 0.3913]	2,670	0.0853	0.8804	0.9513	0.8095
synth_dev_unseen_home_appliance_repair	[-0.0818, 0.0242]	0.2794	[0.1854, 0.3637]	1,720	0.0638	0.8794	0.8779	0.8808
synth_dev_unseen_online_auction_site	[-0.1098, -0.0269]	0.2162	[0.1336, 0.3056]	2,355	0.0686	0.8607	0.9130	0.8084
synth_dev_unseen_student_services_office	[-0.0788, 0.0190]	0.1620	[0.0691, 0.2693]	885	0.0460	0.9167	0.9379	0.8955
synth_dev_unseen_tattoo_studio	[-0.2107, 0.1054]	0.5263	[0.3397, 0.7509]	2,350	0.1146	0.7048	0.7617	0.6479
synth_dev_unseen_vpn_service	[-0.1954, 0.0392]	0.1953	[0.0806, 0.3655]	1,320	0.0386	0.8305	0.7955	0.8655
synth_dev_unseen_waste_collection_service	[-0.1127, 0.1043]	0.4776	[0.3253, 0.6229]	840	0.1695	0.7418	0.9167	0.5670
synth_dev_unseen_yoga_studio	[-0.0575, 0.1138]	0.5399	[0.4328, 0.6467]	2,725	0.0485	0.7966	0.8128	0.7803

information.by_family	check.brier	check.auc	check.right_yes_0.5	check.right_yes_0.2	check.right_yes_0.05	check.ece_p975_twoway	check.ba_p025_twoway	check.ece_interval_twoway
abcd_dev	0.1455	0.8836	0.8913	0.6720	0.3014	0.1643	0.7476	[0.0538, 0.1643]
bitext_dev	0.0358	0.9915	0.9720	0.8966	0.6460	0.1086	0.8822	[0.0351, 0.1086]
clinc_dev	0.0957	0.9789	0.9855	0.9446	0.7821	0.1822	0.8168	[0.0914, 0.1822]
dstc11_banking_guard	0.0424	0.9871	0.9218	0.7466	0.3829	0.0999	0.8865	[0.0461, 0.0999]
dstc11_finance_guard	0.0461	0.9831	0.9660	0.8765	0.5990	0.0754	0.9118	[0.0279, 0.0754]
dstc11_insurance	0.0436	0.9945	0.9078	0.7110	0.3412	0.1213	0.9004	[0.0497, 0.1213]
hint3_dev	0.1553	0.8766	0.9820	0.9318	0.7421	n/a	n/a	[0.0687, 0.4008]
ind_dev_unseen_electricity_grid_operator	0.1062	0.9428	0.8072	0.5114	0.1805	0.1390	0.8048	[0.0527, 0.1390]
ind_dev_unseen_property_management_company	0.0818	0.9580	0.8910	0.6714	0.3007	0.0738	0.8541	[0.0340, 0.0738]
ind_dev_unseen_tax_and_social_security_office	0.0813	0.9709	0.8356	0.5595	0.2110	0.1196	0.8547	[0.0463, 0.1196]
ind_dev_unseen_telecom_operator	0.0886	0.9495	0.9087	0.7133	0.3438	0.1147	0.8192	[0.0400, 0.1147]
sgd_unseen	0.0344	0.9948	1.0000	1.0000	1.0000	n/a	n/a	[0.0369, 0.2205]
synth_dev_seen_domain	0.0483	0.9883	0.9780	0.9174	0.7005	0.0841	0.9139	[0.0435, 0.0841]
synth_dev_unseen_bookstore	0.2920	0.8574	0.5607	0.2419	0.0630	n/a	n/a	[0.0524, 0.4721]
synth_dev_unseen_camera_store	0.1683	0.8607	0.6862	0.3534	0.1032	n/a	n/a	[0.0715, 0.2665]
synth_dev_unseen_car_dealership_service_department	0.0929	0.9538	0.8395	0.5667	0.2159	n/a	n/a	[0.0776, 0.2065]
synth_dev_unseen_car_wash_subscription	0.1118	0.9304	0.7927	0.4887	0.1675	0.1750	0.7655	[0.0731, 0.1750]
synth_dev_unseen_dental_practice	0.1012	0.9379	0.8384	0.5647	0.2145	0.1013	0.7839	[0.0388, 0.1013]
synth_dev_unseen_ebook_platform	0.0899	0.9624	0.8331	0.5552	0.2081	0.1453	0.8318	[0.0470, 0.1453]
synth_dev_unseen_home_appliance_repair	0.0898	0.9489	0.8805	0.6481	0.2794	n/a	n/a	[0.0561, 0.1324]
synth_dev_unseen_online_auction_site	0.1001	0.9460	0.8265	0.5436	0.2005	0.1383	0.8119	[0.0400, 0.1383]
synth_dev_unseen_student_services_office	0.0568	0.9817	0.8997	0.6917	0.3208	0.0903	0.8796	[0.0432, 0.0903]
synth_dev_unseen_tattoo_studio	0.1955	0.8051	0.6839	0.3510	0.1022	0.1861	0.6532	[0.0807, 0.1861]
synth_dev_unseen_vpn_service	0.1132	0.9202	0.8554	0.5966	0.2374	n/a	n/a	[0.0409, 0.2180]
synth_dev_unseen_waste_collection_service	0.1833	0.8782	0.6792	0.3461	0.1002	0.2453	0.6727	[0.1035, 0.2453]
synth_dev_unseen_yoga_studio	0.1441	0.8739	0.7872	0.4805	0.1630	0.1317	0.7350	[0.0411, 0.1317]

information.by_family	check.ba_interval_twoway	check.tyr_interval_twoway	check.tnr_interval_twoway	check.brier_interval_twoway	check.auc_interval_twoway	check.right_yes_0.5_interval_twoway	check.right_yes_0.2_interval_twoway	check.right_yes_0.05_interval_twoway
abcd_dev	[0.7476, 0.8402]	[0.6039, 0.7631]	[0.8738, 0.9490]	[0.1197, 0.1809]	[0.8497, 0.9152]	[0.8451, 0.9312]	[0.5770, 0.7720]	[0.2231, 0.4161]
bitext_dev	[0.8822, 0.9910]	[0.8043, 0.9970]	[0.9372, 0.9984]	[0.0117, 0.0680]	[0.9774, 0.9997]	[0.9357, 0.9984]	[0.7844, 0.9936]	[0.4338, 0.9702]
clinc_dev	[0.8168, 0.9144]	[0.6654, 0.8414]	[0.9762, 0.9968]	[0.0665, 0.1358]	[0.9658, 0.9891]	[0.9704, 0.9957]	[0.8914, 0.9829]	[0.6334, 0.9235]
dstc11_banking_guard	[0.8865, 0.9926]	[0.9635, 1.0000]	[0.7641, 1.0000]	[0.0148, 0.0845]	[0.9594, 0.9999]	[0.8071, 1.0000]	[0.5112, 1.0000]	[0.1804, 1.0000]
dstc11_finance_guard	[0.9118, 0.9670]	[0.8714, 0.9579]	[0.9406, 0.9862]	[0.0282, 0.0695]	[0.9684, 0.9937]	[0.9375, 0.9854]	[0.7895, 0.9441]	[0.4412, 0.7805]
dstc11_insurance	[0.9004, 0.9834]	[0.9834, 1.0000]	[0.7899, 0.9667]	[0.0182, 0.0717]	[0.9842, 0.9993]	[0.8254, 0.9675]	[0.5416, 0.8814]	[0.1992, 0.6102]
hint3_dev	[0.5619, 0.9606]	[0.3529, 0.9402]	[0.9555, 1.0000]	[0.0267, 0.3600]	[0.7109, 0.9992]	[0.9297, 1.0000]	[0.7677, 1.0000]	[0.4103, 1.0000]
ind_dev_unseen_electricity_grid_operator	[0.8048, 0.8977]	[0.8803, 0.9559]	[0.6653, 0.8650]	[0.0762, 0.1365]	[0.9125, 0.9663]	[0.7340, 0.8708]	[0.4083, 0.6276]	[0.1268, 0.2619]
ind_dev_unseen_property_management_company	[0.8541, 0.9202]	[0.8432, 0.9303]	[0.8337, 0.9348]	[0.0632, 0.1052]	[0.9333, 0.9765]	[0.8428, 0.9312]	[0.5728, 0.7719]	[0.2201, 0.4160]
ind_dev_unseen_tax_and_social_security_office	[0.8547, 0.9205]	[0.9329, 0.9850]	[0.7495, 0.8578]	[0.0596, 0.1020]	[0.9541, 0.9835]	[0.7933, 0.8717]	[0.4897, 0.6295]	[0.1681, 0.2635]
ind_dev_unseen_telecom_operator	[0.8192, 0.9260]	[0.7418, 0.9274]	[0.8429, 0.9643]	[0.0596, 0.1286]	[0.9073, 0.9810]	[0.8428, 0.9586]	[0.5727, 0.8526]	[0.2201, 0.5492]
sgd_unseen	[0.8621, 1.0000]	[0.8621, 1.0000]	[0.9999999999999998, 1.0000000000000002]	[0.0026, 0.1185]	[0.9880, 1.0000000000000016]	[0.9999999999999998, 1.0000000000000002]	[0.9999999999999991, 1.0000000000000009]	[0.9999999999999954, 1.0000000000000047]
synth_dev_seen_domain	[0.9139, 0.9597]	[0.8557, 0.9359]	[0.9625, 0.9920]	[0.0347, 0.0655]	[0.9794, 0.9949]	[0.9594, 0.9913]	[0.8553, 0.9662]	[0.5545, 0.8574]
synth_dev_unseen_bookstore	[0.4584, 0.9714]	[0.8966, 0.9714]	[0.2656, 0.2656]	[0.0122, 0.3994]	[0.8425, 0.8710]	[0.5556, 0.5654]	[0.2381, 0.2454]	[0.0617, 0.0641]
synth_dev_unseen_camera_store	[0.6405, 1.0000]	[0.7931, 1.0000]	[0.3529, 0.8706]	[0.0322, 0.2475]	[0.6796, 1.0000]	[0.5577, 0.8793]	[0.2397, 0.6456]	[0.0622, 0.2772]
synth_dev_unseen_car_dealership_service_department	[0.7756, 0.9636]	[0.7991, 0.9755]	[0.5159, 1.0000]	[0.0330, 0.1427]	[0.8790, 0.9962]	[0.6564, 1.0000]	[0.3233, 1.0000]	[0.0914, 1.0000]
synth_dev_unseen_car_wash_subscription	[0.7655, 0.9095]	[0.7826, 0.9706]	[0.5789, 0.9134]	[0.0666, 0.1443]	[0.8748, 0.9762]	[0.6876, 0.9110]	[0.3549, 0.7190]	[0.1038, 0.3501]
synth_dev_unseen_dental_practice	[0.7839, 0.9088]	[0.7763, 0.9396]	[0.7554, 0.8904]	[0.0676, 0.1378]	[0.8875, 0.9735]	[0.7724, 0.8919]	[0.4589, 0.6734]	[0.1515, 0.3027]
synth_dev_unseen_ebook_platform	[0.8318, 0.9343]	[0.9130, 0.9787]	[0.6815, 0.9106]	[0.0528, 0.1257]	[0.9329, 0.9839]	[0.7518, 0.9132]	[0.4310, 0.7245]	[0.1375, 0.3564]
synth_dev_unseen_home_appliance_repair	[0.8340, 0.9187]	[0.8108, 0.9336]	[0.7924, 0.9583]	[0.0687, 0.1171]	[0.9173, 0.9721]	[0.8088, 0.9548]	[0.5140, 0.8409]	[0.1821, 0.5267]
synth_dev_unseen_online_auction_site	[0.8119, 0.9195]	[0.8582, 0.9622]	[0.6740, 0.9167]	[0.0613, 0.1315]	[0.9126, 0.9784]	[0.7411, 0.9157]	[0.4172, 0.7310]	[0.1309, 0.3639]
synth_dev_unseen_student_services_office	[0.8796, 0.9512]	[0.8895, 0.9781]	[0.8158, 0.9513]	[0.0353, 0.0799]	[0.9670, 0.9934]	[0.8358, 0.9502]	[0.5600, 0.8266]	[0.2113, 0.5008]
synth_dev_unseen_tattoo_studio	[0.6532, 0.7637]	[0.6333, 0.8667]	[0.4949, 0.8090]	[0.1548, 0.2289]	[0.7447, 0.8632]	[0.6186, 0.7776]	[0.2885, 0.4664]	[0.0787, 0.1554]
synth_dev_unseen_vpn_service	[0.6715, 0.9326]	[0.6060, 0.9548]	[0.7227, 0.9773]	[0.0559, 0.2083]	[0.8279, 0.9824]	[0.7240, 0.9759]	[0.3961, 0.9101]	[0.1213, 0.6806]
synth_dev_unseen_waste_collection_service	[0.6727, 0.8364]	[0.8314, 0.9810]	[0.4281, 0.6962]	[0.1204, 0.2277]	[0.7988, 0.9423]	[0.6146, 0.7511]	[0.2851, 0.4301]	[0.0774, 0.1371]
synth_dev_unseen_yoga_studio	[0.7350, 0.8504]	[0.7411, 0.8782]	[0.6314, 0.8848]	[0.1107, 0.1829]	[0.8156, 0.9227]	[0.6944, 0.8749]	[0.3623, 0.6361]	[0.1068, 0.2690]

information.by_family	check.interval_resamples_not_finite.tnr	check.interval_resamples_not_finite.auc	check.interval_resamples_not_finite.right_yes_0.5	check.interval_resamples_not_finite.right_yes_0.2	check.interval_resamples_not_finite.right_yes_0.05	scores.interval_resamples_not_finite.ece	scores.interval_resamples_not_finite.cand_ece	scores.interval_resamples_not_finite.rem_ece_6 to 20
abcd_dev
bitext_dev
clinc_dev
dstc11_banking_guard	1	1	1	1	1
dstc11_finance_guard
dstc11_insurance
hint3_dev	10	10	10	10	10	2	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_property_management_company
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_telecom_operator
sgd_unseen	290	290	290	290	290	42	42	42
synth_dev_seen_domain
synth_dev_unseen_bookstore	1,192	1,192	1,192	1,192	1,192	290	290
synth_dev_unseen_camera_store	125	125	125	125	125	9	9	9
synth_dev_unseen_car_dealership_service_department	62	62	62	62	62	5	5	5
synth_dev_unseen_car_wash_subscription	2	2	2	2	2
synth_dev_unseen_dental_practice
synth_dev_unseen_ebook_platform
synth_dev_unseen_home_appliance_repair	3	3	3	3	3	1	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_vpn_service	22	22	22	22	22	1	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio

information.by_family	scores.interval_resamples_not_finite.rem_ece	scores.interval_resamples_not_finite.recall_at_5	scores.interval_resamples_not_finite.accuracy	scores.interval_resamples_not_finite.mean_top	scores.interval_resamples_not_finite.mean_top_minus_accuracy	scores.interval_resamples_not_finite.brier	check.interval_resamples_not_finite.ece	check.interval_resamples_not_finite.ba
abcd_dev
bitext_dev
clinc_dev
dstc11_banking_guard
dstc11_finance_guard
dstc11_insurance
hint3_dev	2	2	2	2	2	2	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_property_management_company
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_telecom_operator
sgd_unseen	42	42	42	42	42	42	42	42
synth_dev_seen_domain
synth_dev_unseen_bookstore		290	290	290	290	290	290	290
synth_dev_unseen_camera_store	9	9	9	9	9	9	9	9
synth_dev_unseen_car_dealership_service_department	5	5	5	5	5	5	5	5
synth_dev_unseen_car_wash_subscription
synth_dev_unseen_dental_practice
synth_dev_unseen_ebook_platform
synth_dev_unseen_home_appliance_repair	1	1	1	1	1	1	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_vpn_service	1	1	1	1	1	1	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio

information.by_family	check.interval_resamples_not_finite.tyr	check.interval_resamples_not_finite.brier
abcd_dev
bitext_dev
clinc_dev
dstc11_banking_guard
dstc11_finance_guard
dstc11_insurance
hint3_dev	2	2
ind_dev_unseen_electricity_grid_operator
ind_dev_unseen_property_management_company
ind_dev_unseen_tax_and_social_security_office
ind_dev_unseen_telecom_operator
sgd_unseen	42	42
synth_dev_seen_domain
synth_dev_unseen_bookstore	290	290
synth_dev_unseen_camera_store	9	9
synth_dev_unseen_car_dealership_service_department	5	5
synth_dev_unseen_car_wash_subscription
synth_dev_unseen_dental_practice
synth_dev_unseen_ebook_platform
synth_dev_unseen_home_appliance_repair	1	1
synth_dev_unseen_online_auction_site
synth_dev_unseen_student_services_office
synth_dev_unseen_tattoo_studio
synth_dev_unseen_vpn_service	1	1
synth_dev_unseen_waste_collection_service
synth_dev_unseen_yoga_studio

information.val.keys	labels	rows	ece	ece_p975	ece_interval	cand_ece	cand_ece_p975	cand_ece_interval
VAL:banking77_train	75	982	0.0820	0.1159	[0.0575, 0.1159]	0.0213	0.0309	[0.0177, 0.0309]
VAL:clinc_bank_validation	29	580	0.0297	0.0452	[0.0261, 0.0452]	0.0107	0.0157	[0.0086, 0.0157]
VAL:hwu64_train	62	877	0.0291	0.0583	[0.0253, 0.0583]	0.0106	0.0201	[0.0095, 0.0201]
VAL:massive_dev@da-DK	57	1,867	0.0420	0.0768	[0.0253, 0.0768]	0.0157	0.0284	[0.0097, 0.0284]
VAL:massive_dev@de-DE	57	1,867	0.0444	0.0819	[0.0262, 0.0819]	0.0172	0.0297	[0.0102, 0.0297]
VAL:massive_dev@el-GR	57	1,867	0.0466	0.0895	[0.0275, 0.0895]	0.0185	0.0335	[0.0118, 0.0335]
VAL:massive_dev@en-US	57	1,867	0.0252	0.0564	[0.0190, 0.0564]	0.0092	0.0182	[0.0065, 0.0182]
VAL:massive_dev@es-ES	57	1,867	0.0425	0.0794	[0.0282, 0.0794]	0.0157	0.0295	[0.0105, 0.0295]
VAL:massive_dev@fi-FI	57	1,867	0.0473	0.0863	[0.0295, 0.0863]	0.0181	0.0309	[0.0115, 0.0309]
VAL:massive_dev@fr-FR	57	1,867	0.0420	0.0801	[0.0255, 0.0801]	0.0183	0.0317	[0.0111, 0.0317]
VAL:massive_dev@hu-HU	57	1,867	0.0592	0.0957	[0.0330, 0.0957]	0.0216	0.0359	[0.0120, 0.0359]
VAL:massive_dev@it-IT	57	1,867	0.0399	0.0747	[0.0251, 0.0747]	0.0161	0.0292	[0.0109, 0.0292]
VAL:massive_dev@lv-LV	57	1,867	0.0396	0.0794	[0.0289, 0.0794]	0.0146	0.0274	[0.0107, 0.0274]
VAL:massive_dev@nb-NO	57	1,867	0.0290	0.0658	[0.0183, 0.0658]	0.0143	0.0271	[0.0083, 0.0271]
VAL:massive_dev@nl-NL	57	1,867	0.0412	0.0753	[0.0268, 0.0753]	0.0168	0.0282	[0.0105, 0.0282]
VAL:massive_dev@pl-PL	57	1,867	0.0398	0.0802	[0.0236, 0.0802]	0.0165	0.0296	[0.0097, 0.0296]
VAL:massive_dev@pt-PT	57	1,867	0.0451	0.0825	[0.0249, 0.0825]	0.0173	0.0309	[0.0106, 0.0309]
VAL:massive_dev@ro-RO	57	1,867	0.0455	0.0844	[0.0291, 0.0844]	0.0184	0.0317	[0.0115, 0.0317]
VAL:massive_dev@ru-RU	57	1,867	0.0349	0.0645	[0.0239, 0.0645]	0.0141	0.0255	[0.0097, 0.0255]
VAL:massive_dev@sl-SL	57	1,867	0.0550	0.0977	[0.0327, 0.0977]	0.0215	0.0364	[0.0130, 0.0364]
VAL:massive_dev@sv-SE	57	1,867	0.0346	0.0673	[0.0252, 0.0673]	0.0159	0.0270	[0.0114, 0.0270]

information.val.keys	rem_ece	rem_ece_p975	rem_ece_bands.6 to 20	rem_ece_bands.21 to 40	rem_ece_bands.41 or more	remainder_info.recall_at_5	remainder_info.over_5.rows	remainder_info.over_5.not_read_share
VAL:banking77_train	0.0684	0.0844	n/a	n/a	0.0684	0.9552	982	0.0448
VAL:clinc_bank_validation	0.0105	0.0188	n/a	0.0105	n/a	0.9948	580	0.0052
VAL:hwu64_train	0.0314	0.0442	n/a	n/a	0.0314	0.9772	877	0.0228
VAL:massive_dev@da-DK	0.0077	0.0199	n/a	n/a	0.0077	0.9561	1,867	0.0439
VAL:massive_dev@de-DE	0.0047	0.0183	n/a	n/a	0.0047	0.9550	1,867	0.0450
VAL:massive_dev@el-GR	0.0145	0.0268	n/a	n/a	0.0145	0.9513	1,867	0.0487
VAL:massive_dev@en-US	0.0076	0.0224	n/a	n/a	0.0076	0.9625	1,867	0.0375
VAL:massive_dev@es-ES	0.0098	0.0241	n/a	n/a	0.0098	0.9523	1,867	0.0477
VAL:massive_dev@fi-FI	0.0160	0.0301	n/a	n/a	0.0160	0.9523	1,867	0.0477
VAL:massive_dev@fr-FR	0.0111	0.0233	n/a	n/a	0.0111	0.9545	1,867	0.0455
VAL:massive_dev@hu-HU	0.0099	0.0245	n/a	n/a	0.0099	0.9497	1,867	0.0503
VAL:massive_dev@it-IT	0.0120	0.0235	n/a	n/a	0.0120	0.9630	1,867	0.0370
VAL:massive_dev@lv-LV	0.0068	0.0219	n/a	n/a	0.0068	0.9395	1,867	0.0605
VAL:massive_dev@nb-NO	0.0145	0.0263	n/a	n/a	0.0145	0.9630	1,867	0.0370
VAL:massive_dev@nl-NL	0.0152	0.0279	n/a	n/a	0.0152	0.9534	1,867	0.0466
VAL:massive_dev@pl-PL	0.0113	0.0239	n/a	n/a	0.0113	0.9614	1,867	0.0386
VAL:massive_dev@pt-PT	0.0116	0.0213	n/a	n/a	0.0116	0.9598	1,867	0.0402
VAL:massive_dev@ro-RO	0.0150	0.0264	n/a	n/a	0.0150	0.9529	1,867	0.0471
VAL:massive_dev@ru-RU	0.0118	0.0214	n/a	n/a	0.0118	0.9630	1,867	0.0370
VAL:massive_dev@sl-SL	0.0145	0.0280	n/a	n/a	0.0145	0.9464	1,867	0.0536
VAL:massive_dev@sv-SE	0.0139	0.0230	n/a	n/a	0.0139	0.9620	1,867	0.0380

information.val.keys	remainder_info.over_5.mean_remainder	remainder_info.over_5.remainder_ece	remainder_info.by_size.6 to 20.not_read_share	remainder_info.by_size.6 to 20.mean_remainder	remainder_info.by_size.6 to 20.remainder_ece	remainder_info.by_size.21 to 40.rows	remainder_info.by_size.21 to 40.not_read_share
VAL:banking77_train	0.1132	0.0684	n/a	n/a	n/a	0	n/a
VAL:clinc_bank_validation	0.0115	0.0105	n/a	n/a	n/a	580	0.0052
VAL:hwu64_train	0.0476	0.0314	n/a	n/a	n/a	0	n/a
VAL:massive_dev@da-DK	0.0472	0.0077	n/a	n/a	n/a	0	n/a
VAL:massive_dev@de-DE	0.0486	0.0047	n/a	n/a	n/a	0	n/a
VAL:massive_dev@el-GR	0.0510	0.0145	n/a	n/a	n/a	0	n/a
VAL:massive_dev@en-US	0.0442	0.0076	n/a	n/a	n/a	0	n/a
VAL:massive_dev@es-ES	0.0484	0.0098	n/a	n/a	n/a	0	n/a
VAL:massive_dev@fi-FI	0.0530	0.0160	n/a	n/a	n/a	0	n/a
VAL:massive_dev@fr-FR	0.0495	0.0111	n/a	n/a	n/a	0	n/a
VAL:massive_dev@hu-HU	0.0528	0.0099	n/a	n/a	n/a	0	n/a
VAL:massive_dev@it-IT	0.0483	0.0120	n/a	n/a	n/a	0	n/a
VAL:massive_dev@lv-LV	0.0575	0.0068	n/a	n/a	n/a	0	n/a
VAL:massive_dev@nb-NO	0.0509	0.0145	n/a	n/a	n/a	0	n/a
VAL:massive_dev@nl-NL	0.0477	0.0152	n/a	n/a	n/a	0	n/a
VAL:massive_dev@pl-PL	0.0492	0.0113	n/a	n/a	n/a	0	n/a
VAL:massive_dev@pt-PT	0.0487	0.0116	n/a	n/a	n/a	0	n/a
VAL:massive_dev@ro-RO	0.0517	0.0150	n/a	n/a	n/a	0	n/a
VAL:massive_dev@ru-RU	0.0478	0.0118	n/a	n/a	n/a	0	n/a
VAL:massive_dev@sl-SL	0.0526	0.0145	n/a	n/a	n/a	0	n/a
VAL:massive_dev@sv-SE	0.0476	0.0139	n/a	n/a	n/a	0	n/a

information.val.keys	remainder_info.by_size.21 to 40.mean_remainder	remainder_info.by_size.21 to 40.remainder_ece	remainder_info.by_size.41 or more.rows	remainder_info.by_size.41 or more.not_read_share	remainder_info.by_size.41 or more.mean_remainder	remainder_info.by_size.41 or more.remainder_ece	rem_ece_band_intervals.6 to 20	rem_ece_band_intervals.21 to 40
VAL:banking77_train	n/a	n/a	982	0.0448	0.1132	0.0684	n/a	n/a
VAL:clinc_bank_validation	0.0115	0.0105	0	n/a	n/a	n/a	n/a	[0.0076, 0.0188]
VAL:hwu64_train	n/a	n/a	877	0.0228	0.0476	0.0314	n/a	n/a
VAL:massive_dev@da-DK	n/a	n/a	1,867	0.0439	0.0472	0.0077	n/a	n/a
VAL:massive_dev@de-DE	n/a	n/a	1,867	0.0450	0.0486	0.0047	n/a	n/a
VAL:massive_dev@el-GR	n/a	n/a	1,867	0.0487	0.0510	0.0145	n/a	n/a
VAL:massive_dev@en-US	n/a	n/a	1,867	0.0375	0.0442	0.0076	n/a	n/a
VAL:massive_dev@es-ES	n/a	n/a	1,867	0.0477	0.0484	0.0098	n/a	n/a
VAL:massive_dev@fi-FI	n/a	n/a	1,867	0.0477	0.0530	0.0160	n/a	n/a
VAL:massive_dev@fr-FR	n/a	n/a	1,867	0.0455	0.0495	0.0111	n/a	n/a
VAL:massive_dev@hu-HU	n/a	n/a	1,867	0.0503	0.0528	0.0099	n/a	n/a
VAL:massive_dev@it-IT	n/a	n/a	1,867	0.0370	0.0483	0.0120	n/a	n/a
VAL:massive_dev@lv-LV	n/a	n/a	1,867	0.0605	0.0575	0.0068	n/a	n/a
VAL:massive_dev@nb-NO	n/a	n/a	1,867	0.0370	0.0509	0.0145	n/a	n/a
VAL:massive_dev@nl-NL	n/a	n/a	1,867	0.0466	0.0477	0.0152	n/a	n/a
VAL:massive_dev@pl-PL	n/a	n/a	1,867	0.0386	0.0492	0.0113	n/a	n/a
VAL:massive_dev@pt-PT	n/a	n/a	1,867	0.0402	0.0487	0.0116	n/a	n/a
VAL:massive_dev@ro-RO	n/a	n/a	1,867	0.0471	0.0517	0.0150	n/a	n/a
VAL:massive_dev@ru-RU	n/a	n/a	1,867	0.0370	0.0478	0.0118	n/a	n/a
VAL:massive_dev@sl-SL	n/a	n/a	1,867	0.0536	0.0526	0.0145	n/a	n/a
VAL:massive_dev@sv-SE	n/a	n/a	1,867	0.0380	0.0476	0.0139	n/a	n/a

information.val.keys	rem_ece_band_intervals.41 or more	rem_ece_interval	recall_at_5	recall_at_5_interval	accuracy	accuracy_interval	mean_top	mean_top_interval
VAL:banking77_train	[0.0542, 0.0844]	[0.0542, 0.0844]	0.9552	[0.9370, 0.9709]	0.8106	[0.7649, 0.8547]	0.7285	[0.6963, 0.7587]
VAL:clinc_bank_validation	n/a	[0.0076, 0.0188]	0.9948	[0.9862, 1.0000]	0.9690	[0.9500, 0.9845]	0.9490	[0.9354, 0.9624]
VAL:hwu64_train	[0.0214, 0.0442]	[0.0214, 0.0442]	0.9772	[0.9624, 0.9884]	0.8575	[0.8193, 0.8922]	0.8508	[0.8203, 0.8788]
VAL:massive_dev@da-DK	[0.0059, 0.0199]	[0.0059, 0.0199]	0.9561	[0.9432, 0.9679]	0.8018	[0.7457, 0.8437]	0.8426	[0.8121, 0.8686]
VAL:massive_dev@de-DE	[0.0050, 0.0183]	[0.0050, 0.0183]	0.9550	[0.9403, 0.9668]	0.8040	[0.7462, 0.8497]	0.8394	[0.8074, 0.8667]
VAL:massive_dev@el-GR	[0.0099, 0.0268]	[0.0099, 0.0268]	0.9513	[0.9378, 0.9633]	0.7933	[0.7357, 0.8381]	0.8364	[0.8083, 0.8594]
VAL:massive_dev@en-US	[0.0065, 0.0224]	[0.0065, 0.0224]	0.9625	[0.9379, 0.9796]	0.8388	[0.7800, 0.8865]	0.8555	[0.8218, 0.8843]
VAL:massive_dev@es-ES	[0.0089, 0.0241]	[0.0089, 0.0241]	0.9523	[0.9352, 0.9672]	0.8061	[0.7459, 0.8533]	0.8432	[0.8131, 0.8675]
VAL:massive_dev@fi-FI	[0.0104, 0.0301]	[0.0104, 0.0301]	0.9523	[0.9409, 0.9622]	0.7975	[0.7391, 0.8455]	0.8340	[0.8049, 0.8575]
VAL:massive_dev@fr-FR	[0.0095, 0.0233]	[0.0095, 0.0233]	0.9545	[0.9376, 0.9681]	0.8034	[0.7449, 0.8499]	0.8446	[0.8154, 0.8688]
VAL:massive_dev@hu-HU	[0.0064, 0.0245]	[0.0064, 0.0245]	0.9497	[0.9378, 0.9615]	0.7734	[0.7135, 0.8217]	0.8286	[0.7972, 0.8551]
VAL:massive_dev@it-IT	[0.0069, 0.0235]	[0.0069, 0.0235]	0.9630	[0.9497, 0.9745]	0.8056	[0.7513, 0.8499]	0.8442	[0.8102, 0.8714]
VAL:massive_dev@lv-LV	[0.0059, 0.0219]	[0.0059, 0.0219]	0.9395	[0.9222, 0.9558]	0.7718	[0.7132, 0.8184]	0.8106	[0.7797, 0.8365]
VAL:massive_dev@nb-NO	[0.0092, 0.0263]	[0.0092, 0.0263]	0.9630	[0.9514, 0.9742]	0.8088	[0.7522, 0.8537]	0.8370	[0.8051, 0.8631]
VAL:massive_dev@nl-NL	[0.0103, 0.0279]	[0.0103, 0.0279]	0.9534	[0.9408, 0.9658]	0.8029	[0.7504, 0.8450]	0.8438	[0.8108, 0.8715]
VAL:massive_dev@pl-PL	[0.0070, 0.0239]	[0.0070, 0.0239]	0.9614	[0.9441, 0.9760]	0.8141	[0.7514, 0.8634]	0.8368	[0.8035, 0.8647]
VAL:massive_dev@pt-PT	[0.0082, 0.0213]	[0.0082, 0.0213]	0.9598	[0.9480, 0.9703]	0.7965	[0.7384, 0.8437]	0.8397	[0.8104, 0.8640]
VAL:massive_dev@ro-RO	[0.0101, 0.0264]	[0.0101, 0.0264]	0.9529	[0.9383, 0.9655]	0.7954	[0.7341, 0.8439]	0.8389	[0.8054, 0.8666]
VAL:massive_dev@ru-RU	[0.0074, 0.0214]	[0.0074, 0.0214]	0.9630	[0.9520, 0.9737]	0.8174	[0.7633, 0.8616]	0.8394	[0.8100, 0.8644]
VAL:massive_dev@sl-SL	[0.0086, 0.0280]	[0.0086, 0.0280]	0.9464	[0.9308, 0.9596]	0.7793	[0.7153, 0.8286]	0.8326	[0.7994, 0.8622]
VAL:massive_dev@sv-SE	[0.01002, 0.0230]	[0.01002, 0.0230]	0.9620	[0.9493, 0.9730]	0.8200	[0.7632, 0.8655]	0.8461	[0.8134, 0.8727]

information.val.keys	mean_top_minus_accuracy	mean_top_minus_accuracy_interval	brier	brier_interval
VAL:banking77_train	-0.0820	[-0.1139, -0.0497]	0.2881	[0.2399, 0.3401]
VAL:clinc_bank_validation	-0.0199	[-0.0341, -0.0053]	0.0444	[0.0243, 0.0677]
VAL:hwu64_train	-0.0067	[-0.0320, 0.0177]	0.2063	[0.1614, 0.2554]
VAL:massive_dev@da-DK	0.0408	[0.0160, 0.0716]	0.2722	[0.2150, 0.3469]
VAL:massive_dev@de-DE	0.0355	[0.0082, 0.0648]	0.2817	[0.2193, 0.3596]
VAL:massive_dev@el-GR	0.0432	[0.0139, 0.0791]	0.2977	[0.2342, 0.3783]
VAL:massive_dev@en-US	0.0167	[-0.0085, 0.0470]	0.2300	[0.1676, 0.3078]
VAL:massive_dev@es-ES	0.0370	[0.0092, 0.0729]	0.2745	[0.2118, 0.3533]
VAL:massive_dev@fi-FI	0.0364	[0.0060, 0.0732]	0.2859	[0.2220, 0.3668]
VAL:massive_dev@fr-FR	0.0412	[0.0119, 0.0760]	0.2799	[0.2163, 0.3592]
VAL:massive_dev@hu-HU	0.0551	[0.0250, 0.0896]	0.3102	[0.2473, 0.3888]
VAL:massive_dev@it-IT	0.0386	[0.0113, 0.0695]	0.2716	[0.2126, 0.3424]
VAL:massive_dev@lv-LV	0.0388	[0.0053, 0.0742]	0.3168	[0.2555, 0.3914]
VAL:massive_dev@nb-NO	0.0282	[0.0024, 0.0578]	0.2691	[0.2097, 0.3431]
VAL:massive_dev@nl-NL	0.0409	[0.0156, 0.0687]	0.2740	[0.2173, 0.3460]
VAL:massive_dev@pl-PL	0.0226	[-0.0063, 0.0583]	0.2707	[0.2034, 0.3543]
VAL:massive_dev@pt-PT	0.0432	[0.0148, 0.0767]	0.2784	[0.2185, 0.3519]
VAL:massive_dev@ro-RO	0.0435	[0.0167, 0.0783]	0.2812	[0.2188, 0.3585]
VAL:massive_dev@ru-RU	0.0221	[-0.0073, 0.0534]	0.2668	[0.2087, 0.3363]
VAL:massive_dev@sl-SL	0.0533	[0.0216, 0.0916]	0.3101	[0.2445, 0.3940]
VAL:massive_dev@sv-SE	0.0261	[0.0004, 0.0563]	0.2581	[0.1985, 0.3317]

information.target.targets	rows	ece	ece_p975	ece_interval	cand_ece	cand_ece_p975	cand_ece_interval	rem_ece
email	4,668	0.0544	0.0774	[0.0374, 0.0774]	0.0154	0.0225	[0.0108, 0.0225]	0.0220
names	4,668	0.0412	0.0700	[0.0268, 0.0700]	0.0133	0.0210	[0.0085, 0.0210]	0.0416
nodia	4,015	0.0470	0.0683	[0.0297, 0.0683]	0.0148	0.0218	[0.0099, 0.0218]	0.0231
typo1	14,004	0.0312	0.0514	[0.0181, 0.0514]	0.0102	0.0162	[0.0062, 0.0162]	0.0104

information.target.targets	rem_ece_p975	rem_ece_bands.6 to 20	rem_ece_bands.21 to 40	rem_ece_bands.41 or more	remainder_info.recall_at_5	remainder_info.over_5.rows	remainder_info.over_5.not_read_share	remainder_info.over_5.mean_remainder
email	0.0359	0.0088	0.0197	0.0604	0.9447	4,627	0.0558	0.0778
names	0.0633	0.0324	0.0574	0.0397	0.9055	4,627	0.0953	0.0577
nodia	0.0388	0.0180	0.0163	0.0505	0.9191	3,990	0.0815	0.1045
typo1	0.0217	0.0141	0.0290	0.0400	0.9291	13,881	0.0715	0.0785

information.target.targets	remainder_info.over_5.remainder_ece	remainder_info.by_size.6 to 20.rows	remainder_info.by_size.6 to 20.not_read_share	remainder_info.by_size.6 to 20.mean_remainder	remainder_info.by_size.6 to 20.remainder_ece	remainder_info.by_size.21 to 40.rows	remainder_info.by_size.21 to 40.not_read_share	remainder_info.by_size.21 to 40.mean_remainder
email	0.0220	1,465	0.0567	0.0492	0.0088	1,223	0.0630	0.0595
names	0.0416	1,465	0.0635	0.0311	0.0324	1,223	0.0957	0.0382
nodia	0.0231	791	0.0784	0.0603	0.0180	864	0.1123	0.0988
typo1	0.0104	4,395	0.0610	0.0471	0.0141	3,669	0.0763	0.0592

information.target.targets	remainder_info.by_size.21 to 40.remainder_ece	remainder_info.by_size.41 or more.rows	remainder_info.by_size.41 or more.not_read_share	remainder_info.by_size.41 or more.mean_remainder	remainder_info.by_size.41 or more.remainder_ece	rem_ece_band_intervals.6 to 20	rem_ece_band_intervals.21 to 40	rem_ece_band_intervals.41 or more
email	0.0197	1,939	0.0505	0.1109	0.0604	[0.0056, 0.0391]	[0.0091, 0.0456]	[0.0403, 0.0795]
names	0.0574	1,939	0.1191	0.0901	0.0397	[0.0121, 0.0582]	[0.0202, 0.1035]	[0.0200, 0.0798]
nodia	0.0163	2,335	0.0711	0.1216	0.0505	[0.0084, 0.0620]	[0.0087, 0.0521]	[0.0325, 0.0688]
typo1	0.0290	5,817	0.0765	0.1144	0.0400	[0.0048, 0.0446]	[0.0110, 0.0567]	[0.0264, 0.0569]

information.target.targets	rem_ece_interval	recall_at_5	recall_at_5_interval	accuracy	accuracy_interval	mean_top	mean_top_interval	mean_top_minus_accuracy
email	[0.0124, 0.0359]	0.9447	[0.9275, 0.9597]	0.7883	[0.7577, 0.8178]	0.7358	[0.7114, 0.7592]	-0.0526
names	[0.0228, 0.0633]	0.9055	[0.8792, 0.9295]	0.7350	[0.6954, 0.7700]	0.7762	[0.7545, 0.7965]	0.0412
nodia	[0.0135, 0.0388]	0.9191	[0.9013, 0.9360]	0.7153	[0.6830, 0.7465]	0.6683	[0.6477, 0.6900]	-0.0470
typo1	[0.0075, 0.0217]	0.9291	[0.9125, 0.9441]	0.7597	[0.7279, 0.7886]	0.7323	[0.7086, 0.7549]	-0.0274

information.target.targets	mean_top_minus_accuracy_interval	brier	brier_interval
email	[-0.0735, -0.0311]	0.3039	[0.2696, 0.3424]
names	[0.0160, 0.0679]	0.3516	[0.3085, 0.3999]
nodia	[-0.0678, -0.0245]	0.3900	[0.3546, 0.4267]
typo1	[-0.0464, -0.0062]	0.3251	[0.2907, 0.3641]

Records

K2 record (E5 (10d)): sha256 07170e1d2152b21faf67826b1253deddd75fde736d980a878e16c5028093c9d3
V-V3S record (E5 (10c)): sha256 13fba8d155424f1169e57b16144f2f0bf443097e46fe80d47d26c20dfeb8d407
beta record of check (tools/newfx_build/newfx_beta.py), on the calibration record of check and scores: sha256 7476fa3cbaf0f7b5dcf9c87996c71d13cb5783ff1a9dc493090a37ab07db3801
calibration file r2intent_calibration.json: sha256 bae5dcdb89ffaf51aad19d396d20460241f47915b32c64af711865e176538285
calibration record of check and scores (PROTOCOL 2026-10-13 00:06): sha256 a7ebb3ff746cc9ac89925d192a1ac9cfd2b943235f8641b6df2c331b5a55e7b0
data_indpack manifest: sha256 988673f5c6638d588cfc352143f3772340bdbaaf75eea5664c7748bc031c8aaa
data_pub manifest: sha256 08f09f914bedd03598112d6a77a9b5b6a7a72a54492b9052de23a4c49bed55f0
data_publevers manifest: sha256 415a51d536112c998292ea21d1398f5881b10ff0718c2ffed456c1c8322fb644
ladder state: sha256 9e39991e8d3c4a6494308c558b5b54bb3a76a8a0df3e2f600106dc820c8d678d
ml_indpack manifest: sha256 843406e08eb9cc8248317dd3446730f55376cf3b827047ddbdb0f0a1d4686968
ml_pub manifest: sha256 20c9a1380120a3274d65d9d1bfe2892fefee7ec78b11c2816714894c6a158283
ml_publevers manifest: sha256 5598a2d9ea6edc3b53c2a1dca62ece14ada7ed36bf94189d0bd68af269ddd548
noise_compare.json (E5 (10b)): sha256 f05d0c1e8422c18a4f3e433811e043facb25a0d49fd0c07d0c773eaaffd90880
rule B record (NW-P against P0): sha256 4da581f6d271d5f07ca414465e8f96194c17e6430c5644314264a57f605c4c9d
shipped record: sha256 f7297b34c6a5294a4e550ce40df06ac8088fb97593123da008498b57b286526f

Downloads last month: -

Safetensors

Model size

0.6B params

Tensor type

F32

F16

Model tree for lukashm/r2intent-v1

Base model

BAAI/bge-reranker-v2-m3

Finetuned

(115)

this model