Clarify blind evaluation and release latency
#3
by aryamantodkar - opened
README.md
CHANGED
|
@@ -109,8 +109,9 @@ voice-agent rows. It did not lead latency.
|
|
| 109 |
Latency used sequential batch-one requests on an Apple M4 MacBook Air with
|
| 110 |
MPS, an optimized Rust extension, and eight Rayon workers. Models were loaded
|
| 111 |
and warmed before request latency was measured. Download and initialization
|
| 112 |
-
are excluded. The
|
| 113 |
-
|
|
|
|
| 114 |
|
| 115 |
See the
|
| 116 |
[`First Evaluation report`](https://github.com/premove-ai/premove-itn/blob/main/eval/voice_agent_itn/results/first-evaluation/REPORT.md)
|
|
@@ -128,7 +129,8 @@ Candidate metadata is generated internally.
|
|
| 128 |
|
| 129 |
- The first use downloads about 1.6 GB; duration depends on the network.
|
| 130 |
- Cached initialization takes several seconds on the tested system.
|
| 131 |
-
- Warm normalization averaged 56.49 ms
|
|
|
|
| 132 |
- Services and transcript streams should keep one normalizer resident.
|
| 133 |
|
| 134 |
## Limitations
|
|
@@ -140,8 +142,8 @@ Candidate metadata is generated internally.
|
|
| 140 |
- The benchmark is synthetic and does not measure live production traffic.
|
| 141 |
- Weaker measured categories include URL, MONEY, CARDINAL, TIME,
|
| 142 |
REFERENCE_ID, and VERSION.
|
| 143 |
-
-
|
| 144 |
-
incomplete.
|
| 145 |
- End-to-end release validation currently covers macOS Apple Silicon, Python
|
| 146 |
3.11, and MPS only.
|
| 147 |
|
|
|
|
| 109 |
Latency used sequential batch-one requests on an Apple M4 MacBook Air with
|
| 110 |
MPS, an optimized Rust extension, and eight Rayon workers. Models were loaded
|
| 111 |
and warmed before request latency was measured. Download and initialization
|
| 112 |
+
are excluded. The Premove timing used the release Rust extension. The backend
|
| 113 |
+
evaluation was blind: each backend received only transcript text.
|
| 114 |
+
Independent human gold adjudication is a separate task and remains pending.
|
| 115 |
|
| 116 |
See the
|
| 117 |
[`First Evaluation report`](https://github.com/premove-ai/premove-itn/blob/main/eval/voice_agent_itn/results/first-evaluation/REPORT.md)
|
|
|
|
| 129 |
|
| 130 |
- The first use downloads about 1.6 GB; duration depends on the network.
|
| 131 |
- Cached initialization takes several seconds on the tested system.
|
| 132 |
+
- Warm normalization averaged 56.49 ms with the retained release Rust build in
|
| 133 |
+
the MPS benchmark.
|
| 134 |
- Services and transcript streams should keep one normalizer resident.
|
| 135 |
|
| 136 |
## Limitations
|
|
|
|
| 142 |
- The benchmark is synthetic and does not measure live production traffic.
|
| 143 |
- Weaker measured categories include URL, MONEY, CARDINAL, TIME,
|
| 144 |
REFERENCE_ID, and VERSION.
|
| 145 |
+
- Independent human gold adjudication and broader contamination checks remain
|
| 146 |
+
incomplete. The backend evaluation itself was blind.
|
| 147 |
- End-to-end release validation currently covers macOS Apple Silicon, Python
|
| 148 |
3.11, and MPS only.
|
| 149 |
|