youval commited on
Commit
2664f88
1 Parent(s): 691d966

update model card

Browse files
Files changed (1) hide show
  1. README.md +65 -54
README.md CHANGED
@@ -1,54 +1,65 @@
1
- ---
2
- language:
3
- - en
4
- ---
5
-
6
- # Model Card for `answer-finder-v1-S-en`
7
-
8
- This model is a question answering model developed by Sinequa. It produces two lists of logit scores corresponding to
9
- the start token and end token of an answer.
10
-
11
- Model name: `answer-finder-v1-S-en`
12
-
13
- ## Supported Languages
14
-
15
- The model was trained and tested in the following languages:
16
-
17
- - English
18
-
19
- ## Scores
20
-
21
- | Metric | Value |
22
- |:--------------------------------------------------------------|-------:|
23
- | F1 Score on SQuAD v2 with Hugging Face evaluation pipeline | 79.4 |
24
- | F1 Score on SQuAD v2 with Haystack evaluation pipeline | 79.5 |
25
-
26
- ## Inference Time
27
-
28
- | GPU Info | Batch size 1 | Batch size 32 |
29
- |:--------------------------------------------------------------|---------------:|---------------:|
30
- | NVIDIA A10 | 4 ms | 44 ms |
31
- | NVIDIA T4 | 7 ms | 128 ms |
32
-
33
- **Note that the Answer Finder models are only used at query time.**
34
-
35
- ## Requirements
36
-
37
- - Minimal Sinequa version: 11.10.0
38
- - GPU memory usage: 560 MiB
39
-
40
- Note that GPU memory usage only includes how much GPU memory the actual model consumes on an NVIDIA T4 GPU with a batch
41
- size of 32. It does not include the fix amount of memory that is consumed by the ONNX Runtime upon initialization which
42
- can be around 0.5 to 1 GiB depending on the used GPU.
43
-
44
- ## Model Details
45
-
46
- ### Overview
47
-
48
- - Number of parameters: 33 million
49
- - Base language model: [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased)
50
- - Insensitive to casing and accents
51
-
52
- ### Training Data
53
-
54
- - [SQuAD v2](https://rajpurkar.github.io/SQuAD-explorer/)
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ ---
5
+
6
+ # Model Card for `answer-finder-v1-S-en`
7
+
8
+ This model is a question answering model developed by Sinequa. It produces two lists of logit scores corresponding to the start token and end token of an answer.
9
+
10
+ Model name: `answer-finder-v1-S-en`
11
+
12
+ ## Supported Languages
13
+
14
+ The model was trained and tested in the following languages:
15
+
16
+ - English
17
+
18
+ ## Scores
19
+
20
+ | Metric | Value |
21
+ |:--------------------------------------------------------------|-------:|
22
+ | F1 Score on SQuAD v2 with Hugging Face evaluation pipeline | 79.4 |
23
+ | F1 Score on SQuAD v2 with Haystack evaluation pipeline | 79.5 |
24
+
25
+ ## Inference Time
26
+
27
+ | GPU | Quantization type | Batch size 1 | Batch size 32 |
28
+ |:------------------------------------------|:------------------|---------------:|---------------:|
29
+ | NVIDIA A10 | FP16 | 1 ms | 10 ms |
30
+ | NVIDIA A10 | FP32 | 3 ms | 43 ms |
31
+ | NVIDIA T4 | FP16 | 2 ms | 22 ms |
32
+ | NVIDIA T4 | FP32 | 5 ms | 130 ms |
33
+ | NVIDIA L4 | FP16 | 2 ms | 12 ms |
34
+ | NVIDIA L4 | FP32 | 5 ms | 62 ms |
35
+
36
+ **Note that the Answer Finder models are only used at query time.**
37
+
38
+ ## Gpu Memory usage
39
+
40
+ | Quantization type | Memory |
41
+ |:-------------------------------------------------|-----------:|
42
+ | FP16 | 300 MiB |
43
+ | FP32 | 550 MiB |
44
+
45
+ Note that GPU memory usage only includes how much GPU memory the actual model consumes on an NVIDIA T4 GPU with a batch
46
+ size of 32. It does not include the fix amount of memory that is consumed by the ONNX Runtime upon initialization which
47
+ can be around 0.5 to 1 GiB depending on the used GPU.
48
+
49
+ ## Requirements
50
+
51
+ - Minimal Sinequa version: 11.10.0
52
+ - Minimal Sinequa version for using FP16 models and GPUs with CUDA compute capability of 8.9+ (like NVIDIA L4): 11.11.0
53
+ - [Cuda compute capability](https://developer.nvidia.com/cuda-gpus): above 5.0 (above 6.0 for FP16 use)
54
+
55
+ ## Model Details
56
+
57
+ ### Overview
58
+
59
+ - Number of parameters: 33 million
60
+ - Base language model: [microsoft/MiniLM-L12-H384-uncased](https://huggingface.co/microsoft/MiniLM-L12-H384-uncased)
61
+ - Insensitive to casing and accents
62
+
63
+ ### Training Data
64
+
65
+ - [SQuAD v2](https://rajpurkar.github.io/SQuAD-explorer/)