Spaces:
Running
Running
Update app.py
Browse files
app.py
CHANGED
@@ -3,59 +3,11 @@ import gradio as gr
|
|
3 |
class ConversionTool:
|
4 |
def __init__(self):
|
5 |
# Initialize widgets
|
6 |
-
self.intro = gr.Markdown(
|
7 |
|
8 |
-
```
|
9 |
-
### # Optimum CLI Export Tool.. tool
|
10 |
-
|
11 |
-
This tool helps organize the process of creating conversion commands when using Intel Optimum for Transformers to convert models outside of a script..
|
12 |
-
|
13 |
-
|
14 |
-
My goal was to make it easier to construct commands for the [Optimum CLI conversion tool](https://huggingface.co/docs/optimum/main/en/intel/openvino/export) which enables converting models to the OpenVINO Intermediate Representation
|
15 |
-
outside of the from.pretrained method used in Transformers with the OpenVINO related classes like OVModelForCausalLM, OVModelForSeq2SeqLM, OVModelForQuestionAnswering, etc.
|
16 |
-
|
17 |
-
## Usage
|
18 |
-
Here I'm assuming you have followed the instructions in the documentation and have all your dependencies in order.
|
19 |
-
|
20 |
-
Run to to get the latest.
|
21 |
-
```
|
22 |
-
pip install --upgrade --upgrade-strategy eager optimum[openvino]
|
23 |
-
```
|
24 |
-
|
25 |
-
Intended workflow:
|
26 |
-
-Select parameters.
|
27 |
-
-Copy command.
|
28 |
-
-Execute in your environment.
|
29 |
-
|
30 |
-
Note: Converstion can take a while.
|
31 |
-
Expect slow performance and rejoice. After all, OpenVINO supports Intel CPUs from 6th gen forward, so you can
|
32 |
-
squeeze performance
|
33 |
-
|
34 |
-
## Discussion
|
35 |
-
|
36 |
-
Leveraging hardware acceleration from OpenVINO requires converting a model into an Intermediate format derived from ONNX. Basically the command we execute rebuilds the model graph from it's source to be optimized for how OpenVINO uses this graph in memory.
|
37 |
-
|
38 |
-
Using OpenVINO effectively requires considering facts about what hardware the Intel PC running the code has available to it alongside details about the model you're working with.
|
39 |
-
|
40 |
-
|
41 |
-
|
42 |
-
|
43 |
-
So here are some questions you should be able to answer before using this tool;
|
44 |
-
- What data types does my CPU support?
|
45 |
-
- What instruction sets?
|
46 |
-
- How will I be using the model?
|
47 |
-
- Do I have enough system memory for this task?
|
48 |
-
|
49 |
-
Visit the [Intel Ark ]([Intel® Processors for PC, Laptops, Servers, and AI | Intel®](https://www.intel.com/content/www/us/en/products/details/processors.html)) product database to find this information. It's *the* ground truth on these sorts of specs. Even so, when testing with different model architecture
|
50 |
-
|
51 |
-
|
52 |
|
53 |
|
54 |
|
55 |
-
|
56 |
-
""")
|
57 |
-
|
58 |
-
|
59 |
self.model_input = gr.Textbox(
|
60 |
label='Model',
|
61 |
placeholder='Model ID on huggingface.co or path on disk',
|
@@ -301,4 +253,55 @@ Visit the [Intel Ark ]([Intel® Processors for PC, Laptops, Servers, and AI | I
|
|
301 |
if __name__ == "__main__":
|
302 |
tool = ConversionTool()
|
303 |
app = tool.gradio_app()
|
304 |
-
app.launch(share = False)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
class ConversionTool:
|
4 |
def __init__(self):
|
5 |
# Initialize widgets
|
6 |
+
self.intro = gr.Markdown(INTRODUCTION)
|
7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
8 |
|
9 |
|
10 |
|
|
|
|
|
|
|
|
|
11 |
self.model_input = gr.Textbox(
|
12 |
label='Model',
|
13 |
placeholder='Model ID on huggingface.co or path on disk',
|
|
|
253 |
if __name__ == "__main__":
|
254 |
tool = ConversionTool()
|
255 |
app = tool.gradio_app()
|
256 |
+
app.launch(share = False)
|
257 |
+
|
258 |
+
|
259 |
+
|
260 |
+
INTRODUCTION="""
|
261 |
+
|
262 |
+
### # Optimum CLI Export Tool.. tool
|
263 |
+
|
264 |
+
This tool helps organize conversion commands when using Intel Optimum for Transformers and respects the order of positional arguments. Otherwise these commands can get quite nuanced to keep track of.
|
265 |
+
|
266 |
+
My goal was to make it easier to construct commands for the [Optimum CLI conversion tool](https://huggingface.co/docs/optimum/main/en/intel/openvino/export) which enables converting models to the OpenVINO Intermediate Representation
|
267 |
+
outside of the from.pretrained method used in Transformers with OpenVINO related classes like OVModelForCausalLM, OVModelForSeq2SeqLM, OVModelForQuestionAnswering, etc, which interface with the OpenVINO runtime.
|
268 |
+
|
269 |
+
## Usage
|
270 |
+
Here I'm assuming you have followed the instructions in the documentation and have all your dependencies in order.
|
271 |
+
|
272 |
+
Run to to get the latest version of the neccessary extension for optimum:
|
273 |
+
```
|
274 |
+
pip install --upgrade --upgrade-strategy eager optimum[openvino]
|
275 |
+
```
|
276 |
+
|
277 |
+
Intended workflow:
|
278 |
+
-Select conversion parameters.
|
279 |
+
-Hit "Submit"
|
280 |
+
-Copy command.
|
281 |
+
-Execute in your environment.
|
282 |
+
|
283 |
+
Note: Converstion can take a while and will be resource intensive.
|
284 |
+
|
285 |
+
|
286 |
+
OpenVINO supports Intel CPUs from 6th gen forward, so you can squeeze performance out of older hardware with
|
287 |
+
different accuracy/performance tradeoffs than the popular quants of GGUFs.
|
288 |
+
|
289 |
+
## Discussion
|
290 |
+
|
291 |
+
Leveraging CPU, GPU and NPU hardware acceleration from OpenVINO requires converting a model into an Intermediate format derived from ONNX.
|
292 |
+
The command we execute rebuilds the model graph from it's source to be optimized for how OpenVINO uses this graph in memory.
|
293 |
+
|
294 |
+
Using OpenVINO effectively requires considering facts about your Intel hardware. Visit the [Intel Ark ]([Intel® Processors for PC, Laptops, Servers, and AI | Intel®](https://www.intel.com/content/www/us/en/products/details/processors.html)) product database to find this information.
|
295 |
+
|
296 |
+
Here are some hardware questions you should be able to answer before using this tool;
|
297 |
+
|
298 |
+
- What data types does my CPU support?
|
299 |
+
- What instruction sets?
|
300 |
+
- How will I be using the model?
|
301 |
+
- Do I have enough system memory for this task?
|
302 |
+
|
303 |
+
|
304 |
+
|
305 |
+
It's *the* ground truth for Intel Hardware specs. Even so, when testing with different model architectures
|
306 |
+
|
307 |
+
"""
|