Document open-weight release and public loading API
#1
by aryaman20 - opened
README.md
CHANGED
|
@@ -23,23 +23,34 @@ utterances. This release is the first frozen, inference-only artifact for its
|
|
| 23 |
structured-value contextual scorer. It ranks deterministic Rust candidates
|
| 24 |
using sentence context, then applies exact maximum-score decoding.
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
The artifact contains the complete trained parameters in `model.safetensors`,
|
| 27 |
the exact DeBERTa configuration, and the tokenizer files used by the selected
|
| 28 |
production checkpoint. Optimizer state, scheduler state, training counters,
|
| 29 |
training data, and evaluation rows are not included.
|
| 30 |
|
| 31 |
The artifact is loaded through the `premove-itn` package. It is not a generic
|
| 32 |
-
Transformers model; use
|
| 33 |
-
stay aligned with the release implementation:
|
| 34 |
|
| 35 |
```python
|
| 36 |
-
from premove_itn
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
|
|
|
|
|
|
| 41 |
```
|
| 42 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
The model uses `microsoft/deberta-v3-large` at the exact revision recorded in
|
| 44 |
`provenance.json`. The artifact includes the tokenizer and base configuration;
|
| 45 |
the base model is not separately redistributed. Microsoft lists the base model
|
|
@@ -64,6 +75,15 @@ The supported span kinds are listed in `config.json`: `DIGIT_SEQUENCE`,
|
|
| 64 |
`CARDINAL`, `TIME`, `DATE`, `MONEY`, `DECIMAL`, `PHONE`, `ELECTRONIC`,
|
| 65 |
`MEASUREMENT`, `ORDINAL`, `PUNCTUATION`, `WHITELIST`, and `WORD`.
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## First Evaluation
|
| 68 |
|
| 69 |
The retained First Evaluation used 1,500 frozen VoiceAgent ITN rows and three
|
|
@@ -85,7 +105,8 @@ was not blind.
|
|
| 85 |
## Limitations
|
| 86 |
|
| 87 |
- This custom scorer is not directly loadable with `AutoModel.from_pretrained`.
|
| 88 |
-
Use `
|
|
|
|
| 89 |
- The artifact is English-only and depends on the package's candidate graph and
|
| 90 |
decoder for end-to-end normalization.
|
| 91 |
- Benchmark results are not a guarantee for unseen domains or formatting
|
|
@@ -99,9 +120,12 @@ was not blind.
|
|
| 99 |
- Package version: `0.1.0`
|
| 100 |
- Hub repository: `premove-itn/premove-itn`
|
| 101 |
- Hub revision: `v0.1.0`
|
|
|
|
|
|
|
|
|
|
| 102 |
- Source checkpoint SHA-256: `9021fa11a028faefb31ef67878170cbe29ed25e68a9a78999f37b120c2ad00d5`
|
| 103 |
- Inference artifact SHA-256: `119c0f19767b61446e04da1f8f01a001edf97a47a66965e7146db2483b4937a1`
|
| 104 |
|
| 105 |
## Licensing
|
| 106 |
|
| 107 |
-
Premove ITN is MIT licensed. The pinned base model is `microsoft/deberta-v3-large`, whose model card lists an MIT license. Review and retain both notices when redistributing this artifact.
|
|
|
|
| 23 |
structured-value contextual scorer. It ranks deterministic Rust candidates
|
| 24 |
using sentence context, then applies exact maximum-score decoding.
|
| 25 |
|
| 26 |
+
**Premove ITN is released as an open-weight contextual inverse text
|
| 27 |
+
normalization model. The inference code and model weights are licensed under
|
| 28 |
+
MIT.**
|
| 29 |
+
|
| 30 |
The artifact contains the complete trained parameters in `model.safetensors`,
|
| 31 |
the exact DeBERTa configuration, and the tokenizer files used by the selected
|
| 32 |
production checkpoint. Optimizer state, scheduler state, training counters,
|
| 33 |
training data, and evaluation rows are not included.
|
| 34 |
|
| 35 |
The artifact is loaded through the `premove-itn` package. It is not a generic
|
| 36 |
+
Transformers model; use `PremoveITN.from_pretrained()` so candidate generation,
|
| 37 |
+
scoring, and decoding stay aligned with the release implementation:
|
| 38 |
|
| 39 |
```python
|
| 40 |
+
from premove_itn import PremoveITN
|
| 41 |
|
| 42 |
+
itn = PremoveITN.from_pretrained(
|
| 43 |
+
"premove-itn/premove-itn",
|
| 44 |
+
revision="v0.1.0",
|
| 45 |
+
)
|
| 46 |
+
print(itn.normalize("call me at four thirty")) # call me at 04:30
|
| 47 |
```
|
| 48 |
|
| 49 |
+
Source code, deterministic Rust realization rules, the Python API, benchmark
|
| 50 |
+
code, and retained results are available in the
|
| 51 |
+
[`premove-ai/premove-itn`](https://github.com/premove-ai/premove-itn)
|
| 52 |
+
repository.
|
| 53 |
+
|
| 54 |
The model uses `microsoft/deberta-v3-large` at the exact revision recorded in
|
| 55 |
`provenance.json`. The artifact includes the tokenizer and base configuration;
|
| 56 |
the base model is not separately redistributed. Microsoft lists the base model
|
|
|
|
| 75 |
`CARDINAL`, `TIME`, `DATE`, `MONEY`, `DECIMAL`, `PHONE`, `ELECTRONIC`,
|
| 76 |
`MEASUREMENT`, `ORDINAL`, `PUNCTUATION`, `WHITELIST`, and `WORD`.
|
| 77 |
|
| 78 |
+
## Runtime
|
| 79 |
+
|
| 80 |
+
The contextual API requires Python 3.11 or newer, PyTorch, Transformers,
|
| 81 |
+
SentencePiece, safetensors, and huggingface_hub. `device="auto"` selects CUDA,
|
| 82 |
+
then Apple MPS, then CPU. Pass `device="cpu"`, `device="mps"`, or
|
| 83 |
+
`device="cuda"` to select a device explicitly. The retained latency result was
|
| 84 |
+
measured with batch size one on Apple MPS. Other runtimes and devices can have
|
| 85 |
+
different latency and memory use.
|
| 86 |
+
|
| 87 |
## First Evaluation
|
| 88 |
|
| 89 |
The retained First Evaluation used 1,500 frozen VoiceAgent ITN rows and three
|
|
|
|
| 105 |
## Limitations
|
| 106 |
|
| 107 |
- This custom scorer is not directly loadable with `AutoModel.from_pretrained`.
|
| 108 |
+
Use `PremoveITN.from_pretrained()` and the matching `premove-itn` package
|
| 109 |
+
version.
|
| 110 |
- The artifact is English-only and depends on the package's candidate graph and
|
| 111 |
decoder for end-to-end normalization.
|
| 112 |
- Benchmark results are not a guarantee for unseen domains or formatting
|
|
|
|
| 120 |
- Package version: `0.1.0`
|
| 121 |
- Hub repository: `premove-itn/premove-itn`
|
| 122 |
- Hub revision: `v0.1.0`
|
| 123 |
+
- Hub commit: `80bda5e2e1fe9542aa628597090242df57c1a157`
|
| 124 |
+
- Base model: `microsoft/deberta-v3-large`
|
| 125 |
+
- Base model revision: `64a8c8eab3e352a784c658aef62be1662607476f`
|
| 126 |
- Source checkpoint SHA-256: `9021fa11a028faefb31ef67878170cbe29ed25e68a9a78999f37b120c2ad00d5`
|
| 127 |
- Inference artifact SHA-256: `119c0f19767b61446e04da1f8f01a001edf97a47a66965e7146db2483b4937a1`
|
| 128 |
|
| 129 |
## Licensing
|
| 130 |
|
| 131 |
+
Premove ITN is MIT licensed. The pinned base model is [`microsoft/deberta-v3-large`](https://huggingface.co/microsoft/deberta-v3-large), whose model card lists an MIT license. The architecture derives from [DeBERTaV3](https://arxiv.org/abs/2111.09543). Review and retain both notices when redistributing this artifact.
|