Clarify blind evaluation and release latency

#3
Files changed (1) hide show
  1. README.md +7 -5
README.md CHANGED
@@ -109,8 +109,9 @@ voice-agent rows. It did not lead latency.
109
  Latency used sequential batch-one requests on an Apple M4 MacBook Air with
110
  MPS, an optimized Rust extension, and eight Rayon workers. Models were loaded
111
  and warmed before request latency was measured. Download and initialization
112
- are excluded. The result is not an estimate of production-traffic accuracy,
113
- and the evaluation was not blind.
 
114
 
115
  See the
116
  [`First Evaluation report`](https://github.com/premove-ai/premove-itn/blob/main/eval/voice_agent_itn/results/first-evaluation/REPORT.md)
@@ -128,7 +129,8 @@ Candidate metadata is generated internally.
128
 
129
  - The first use downloads about 1.6 GB; duration depends on the network.
130
  - Cached initialization takes several seconds on the tested system.
131
- - Warm normalization averaged 56.49 ms in the retained MPS benchmark.
 
132
  - Services and transcript streams should keep one normalizer resident.
133
 
134
  ## Limitations
@@ -140,8 +142,8 @@ Candidate metadata is generated internally.
140
  - The benchmark is synthetic and does not measure live production traffic.
141
  - Weaker measured categories include URL, MONEY, CARDINAL, TIME,
142
  REFERENCE_ID, and VERSION.
143
- - Blind human gold adjudication and broader contamination checks remain
144
- incomplete.
145
  - End-to-end release validation currently covers macOS Apple Silicon, Python
146
  3.11, and MPS only.
147
 
 
109
  Latency used sequential batch-one requests on an Apple M4 MacBook Air with
110
  MPS, an optimized Rust extension, and eight Rayon workers. Models were loaded
111
  and warmed before request latency was measured. Download and initialization
112
+ are excluded. The Premove timing used the release Rust extension. The backend
113
+ evaluation was blind: each backend received only transcript text.
114
+ Independent human gold adjudication is a separate task and remains pending.
115
 
116
  See the
117
  [`First Evaluation report`](https://github.com/premove-ai/premove-itn/blob/main/eval/voice_agent_itn/results/first-evaluation/REPORT.md)
 
129
 
130
  - The first use downloads about 1.6 GB; duration depends on the network.
131
  - Cached initialization takes several seconds on the tested system.
132
+ - Warm normalization averaged 56.49 ms with the retained release Rust build in
133
+ the MPS benchmark.
134
  - Services and transcript streams should keep one normalizer resident.
135
 
136
  ## Limitations
 
142
  - The benchmark is synthetic and does not measure live production traffic.
143
  - Weaker measured categories include URL, MONEY, CARDINAL, TIME,
144
  REFERENCE_ID, and VERSION.
145
+ - Independent human gold adjudication and broader contamination checks remain
146
+ incomplete. The backend evaluation itself was blind.
147
  - End-to-end release validation currently covers macOS Apple Silicon, Python
148
  3.11, and MPS only.
149