Spaces:
Runtime error
Runtime error
File size: 49,272 Bytes
db5855f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 |
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "ef2ed242-3561-464c-8d1c-cc3862e23702",
"metadata": {},
"source": [
"# Instruction following using Databricks Dolly 2.0 and OpenVINO\n",
"\n",
"The instruction following is one of the cornerstones of the current generation of large language models(LLMs). Reinforcement learning with human preferences ([RLHF](https://arxiv.org/abs/1909.08593)) and techniques such as [InstructGPT](https://arxiv.org/abs/2203.02155) has been the core foundation of breakthroughs such as ChatGPT and GPT-4. However, these powerful models remain hidden behind APIs and we know very little about their underlying architecture. Instruction-following models are capable of generating text in response to prompts and are often used for tasks like writing assistance, chatbots, and content generation. Many users now interact with these models regularly and even use them for work but the majority of such models remain closed-source and require massive amounts of computational resources to experiment with.\n",
"\n",
"[Dolly 2.0](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) is the first open-source, instruction-following LLM fine-tuned by Databricks on a transparent and freely available dataset that is also open-sourced to use for commercial purposes. That means Dolly 2.0 is available for commercial applications without the need to pay for API access or share data with third parties. Dolly 2.0 exhibits similar characteristics so ChatGPT despite being much smaller.\n",
"\n",
"In this tutorial, we consider how to run an instruction-following text generation pipeline using Dolly 2.0 and OpenVINO. We will use a pre-trained model from the [Hugging Face Transformers](https://huggingface.co/docs/transformers/index) library. To simplify the user experience, the [Hugging Face Optimum Intel](https://huggingface.co/docs/optimum/intel/index) library is used to convert the models to OpenVINO™ IR format.\n",
"\n",
"The tutorial consists of the following steps:\n",
"\n",
"- Install prerequisites\n",
"- Download and convert the model from a public source using the [OpenVINO integration with Hugging Face Optimum](https://huggingface.co/blog/openvino).\n",
"- Compress model weights to INT8 with [OpenVINO NNCF](https://github.com/openvinotoolkit/nncf)\n",
"- Create an instruction-following inference pipeline\n",
"- Run instruction-following pipeline\n",
"\n",
"\n",
"## About Dolly 2.0\n",
"\n",
"Dolly 2.0 is an instruction-following large language model trained on the Databricks machine-learning platform that is licensed for commercial use. It is based on [Pythia](https://github.com/EleutherAI/pythia) and is trained on ~15k instruction/response fine-tuning records generated by Databricks employees in various capability domains, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.\n",
"Dolly 2.0 works by processing natural language instructions and generating responses that follow the given instructions. It can be used for a wide range of applications, including closed question-answering, summarization, and generation.\n",
"\n",
"The model training process was inspired by [InstructGPT](https://arxiv.org/abs/2203.02155). To train InstructGPT models, the core technique is reinforcement learning from human feedback (RLHF), This technique uses human preferences as a reward signal to fine-tune models, which is important as the safety and alignment problems required to be solved are complex and subjective, and aren’t fully captured by simple automatic metrics. More details about the InstructGPT approach can be found in OpenAI [blog post](https://openai.com/research/instruction-following) \n",
"The breakthrough discovered with InstructGPT is that language models don’t need larger and larger training sets. By using human-evaluated question-and-answer training, authors were able to train a better language model using one hundred times fewer parameters than the previous model. Databricks used a similar approach to create a prompt and response dataset called they call [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k), a corpus of more than 15,000 records generated by thousands of Databricks employees to enable large language models to exhibit the magical interactivity of InstructGPT. More details about the model and dataset can be found in [Databricks blog post](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) and [repo](https://github.com/databrickslabs/dolly)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f97c435a",
"metadata": {},
"source": [
"\n",
"\n",
"\n",
"\n",
"#### Table of contents:\n",
"\n",
"- [Prerequisites](#Prerequisites)\n",
"- [Convert model using Optimum-CLI tool](#Convert-model-using-Optimum-CLI-tool)\n",
"- [Compress model weights](#Compress-model-weights)\n",
" - [Weights Compression using Optimum-CLI](#Weights-Compression-using-Optimum-CLI)\n",
"- [Select model variant and inference device](#Select-model-variant-and-inference-device)\n",
"- [Instantiate Model using Optimum Intel](#Instantiate-Model-using-Optimum-Intel)\n",
"- [Create an instruction-following inference pipeline](#Create-an-instruction-following-inference-pipeline)\n",
" - [Setup imports](#Setup-imports)\n",
" - [Prepare template for user prompt](#Prepare-template-for-user-prompt)\n",
" - [Helpers for output parsing](#Helpers-for-output-parsing)\n",
" - [Main generation function](#Main-generation-function)\n",
" - [Helpers for application](#Helpers-for-application)\n",
"- [Run instruction-following pipeline](#Run-instruction-following-pipeline)\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "08aa16b1-d2f6-4a3a-abfb-5ec278133c80",
"metadata": {},
"source": [
"## Prerequisites\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"First, we should install the [Hugging Face Optimum](https://huggingface.co/docs/optimum/installation) library accelerated by OpenVINO integration.\n",
"The Hugging Face Optimum Intel API is a high-level API that enables us to convert and quantize models from the Hugging Face Transformers library to the OpenVINO™ IR format. For more details, refer to the [Hugging Face Optimum Intel documentation](https://huggingface.co/docs/optimum/intel/inference)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "4421fc85-bed6-4a62-b8fa-19c7ba474891",
"metadata": {},
"outputs": [],
"source": [
"%pip install -Uq pip\n",
"%pip uninstall -q -y optimum optimum-intel\n",
"%pip install --pre -Uq openvino openvino-tokenizers[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly\n",
"%pip install -q \"diffusers>=0.16.1\" \"transformers>=4.33.0\" \"torch>=2.1\" \"nncf>=2.10.0\" onnx \"gradio>=4.19\" --extra-index-url https://download.pytorch.org/whl/cpu\n",
"%pip install -q \"git+https://github.com/huggingface/optimum-intel.git\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e145f4ab-1753-4eec-a369-575a93448462",
"metadata": {
"tags": []
},
"source": [
"## Convert model using Optimum-CLI tool\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"🤗 [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) is the interface between the 🤗 [Transformers](https://huggingface.co/docs/transformers/index) and [Diffusers](https://huggingface.co/docs/diffusers/index) libraries and OpenVINO to accelerate end-to-end pipelines on Intel architectures. It provides ease-to-use cli interface for exporting models to [OpenVINO Intermediate Representation (IR)](https://docs.openvino.ai/2024/documentation/openvino-ir-format.html) format.\n",
"\n",
"The command bellow demonstrates basic command for model export with `optimum-cli`\n",
"\n",
"```bash\n",
"optimum-cli export openvino --model <model_id_or_path> --task <task> <out_dir>\n",
"```\n",
"\n",
"where `--model` argument is model id from HuggingFace Hub or local directory with model (saved using `.save_pretrained` method), `--task ` is one of [supported task](https://huggingface.co/docs/optimum/exporters/task_manager) that exported model should solve. For LLMs it will be `text-generation-with-past`. If model initialization requires to use remote code, `--trust-remote-code` flag additionally should be passed."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "a38d7f46-1a27-48ea-9064-170cd6fdeb5f",
"metadata": {},
"source": [
"## Compress model weights\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"\n",
"The [Weights Compression](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html) algorithm is aimed at compressing the weights of the models and can be used to optimize the model footprint and performance of large models where the size of weights is relatively larger than the size of activations, for example, Large Language Models (LLM). Compared to INT8 compression, INT4 compression improves performance even more, but introduces a minor drop in prediction quality.\n",
"\n",
"\n",
"### Weights Compression using Optimum-CLI\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"You can also apply fp16, 8-bit or 4-bit weight compression on the Linear, Convolutional and Embedding layers when exporting your model with the CLI by setting `--weight-format` to respectively fp16, int8 or int4. This type of optimization allows to reduce the memory footprint and inference latency.\n",
"By default the quantization scheme for int8/int4 will be [asymmetric](https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#asymmetric-quantization), to make it [symmetric](https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Quantization.md#symmetric-quantization) you can add `--sym`.\n",
"\n",
"For INT4 quantization you can also specify the following arguments :\n",
"- The `--group-size` parameter will define the group size to use for quantization, -1 it will results in per-column quantization.\n",
"- The `--ratio` parameter controls the ratio between 4-bit and 8-bit quantization. If set to 0.9, it means that 90% of the layers will be quantized to int4 while 10% will be quantized to int8.\n",
"\n",
"Smaller group_size and ratio values usually improve accuracy at the sacrifice of the model size and inference latency.\n",
"\n",
">**Note**: There may be no speedup for INT4/INT8 compressed models on dGPU."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b30a0744-0a25-47f0-a2c5-9a7822131034",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ec46027bc7f04402a939d0d8de46e115",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Checkbox(value=True, description='Prepare INT4 model')"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "639eb1acfa314f778132b0466163ef38",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Checkbox(value=False, description='Prepare INT8 model')"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "332ca84dafac474ebeb9b9b6f89a70ec",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Checkbox(value=False, description='Prepare FP16 model')"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from IPython.display import Markdown, display\n",
"import ipywidgets as widgets\n",
"\n",
"prepare_int4_model = widgets.Checkbox(\n",
" value=True,\n",
" description=\"Prepare INT4 model\",\n",
" disabled=False,\n",
")\n",
"prepare_int8_model = widgets.Checkbox(\n",
" value=False,\n",
" description=\"Prepare INT8 model\",\n",
" disabled=False,\n",
")\n",
"prepare_fp16_model = widgets.Checkbox(\n",
" value=False,\n",
" description=\"Prepare FP16 model\",\n",
" disabled=False,\n",
")\n",
"\n",
"display(prepare_int4_model)\n",
"display(prepare_int8_model)\n",
"display(prepare_fp16_model)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5dc61e85-176f-455c-be36-6001cbc17a30",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"model_id = \"databricks/dolly-v2-3b\"\n",
"model_path = Path(\"dolly-v2-3b\")\n",
"\n",
"fp16_model_dir = model_path / \"FP16\"\n",
"int8_model_dir = model_path / \"INT8_compressed_weights\"\n",
"int4_model_dir = model_path / \"INT4_compressed_weights\"\n",
"\n",
"\n",
"def convert_to_fp16():\n",
" if (fp16_model_dir / \"openvino_model.xml\").exists():\n",
" return\n",
" fp16_model_dir.mkdir(parents=True, exist_ok=True)\n",
" export_command_base = \"optimum-cli export openvino --model {} --task text-generation-with-past --weight-format fp16\".format(model_id)\n",
" export_command = export_command_base + \" \" + str(fp16_model_dir)\n",
" display(Markdown(\"**Export command:**\"))\n",
" display(Markdown(f\"`{export_command}`\"))\n",
" ! $export_command\n",
"\n",
"\n",
"def convert_to_int8():\n",
" if (int8_model_dir / \"openvino_model.xml\").exists():\n",
" return\n",
" int8_model_dir.mkdir(parents=True, exist_ok=True)\n",
" export_command_base = \"optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int8\".format(model_id)\n",
" export_command = export_command_base + \" \" + str(int8_model_dir)\n",
" display(Markdown(\"**Export command:**\"))\n",
" display(Markdown(f\"`{export_command}`\"))\n",
" ! $export_command\n",
"\n",
"\n",
"def convert_to_int4():\n",
" if (int4_model_dir / \"openvino_model.xml\").exists():\n",
" return\n",
" int4_model_dir.mkdir(parents=True, exist_ok=True)\n",
" export_command_base = \"optimum-cli export openvino --model {} --task text-generation-with-past --weight-format int4\".format(model_id)\n",
" export_command = export_command_base + \" \" + str(int4_model_dir)\n",
" display(Markdown(\"**Export command:**\"))\n",
" display(Markdown(f\"`{export_command}`\"))\n",
" ! $export_command\n",
"\n",
"\n",
"if prepare_fp16_model.value:\n",
" convert_to_fp16()\n",
"if prepare_int8_model.value:\n",
" convert_to_int8()\n",
"if prepare_int4_model.value:\n",
" convert_to_int4()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b65fb768-6125-4426-870b-07ebc1a94c07",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Size of model with INT4 compressed weights is 2154.54 MB\n"
]
}
],
"source": [
"fp16_weights = fp16_model_dir / \"openvino_model.bin\"\n",
"int8_weights = int8_model_dir / \"openvino_model.bin\"\n",
"int4_weights = int4_model_dir / \"openvino_model.bin\"\n",
"\n",
"if fp16_weights.exists():\n",
" print(f\"Size of FP16 model is {fp16_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
"for precision, compressed_weights in zip([8, 4], [int8_weights, int4_weights]):\n",
" if compressed_weights.exists():\n",
" print(f\"Size of model with INT{precision} compressed weights is {compressed_weights.stat().st_size / 1024 / 1024:.2f} MB\")\n",
" if compressed_weights.exists() and fp16_weights.exists():\n",
" print(f\"Compression rate for INT{precision} model: {fp16_weights.stat().st_size / compressed_weights.stat().st_size:.3f}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "367f84f8-33e8-4ad6-bd40-e6fd41d2d703",
"metadata": {},
"source": [
"### Select model variant and inference device\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"select device from dropdown list for running inference using OpenVINO"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f59ee1bc-029f-45f3-90ba-a29db4ce7a1c",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "31e97b72a4e346ed856bad6a24fbf164",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Dropdown(description='Model to run:', options=('INT4',), value='INT4')"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"available_models = []\n",
"if int4_model_dir.exists():\n",
" available_models.append(\"INT4\")\n",
"if int8_model_dir.exists():\n",
" available_models.append(\"INT8\")\n",
"if fp16_model_dir.exists():\n",
" available_models.append(\"FP16\")\n",
"\n",
"model_to_run = widgets.Dropdown(\n",
" options=available_models,\n",
" value=available_models[0],\n",
" description=\"Model to run:\",\n",
" disabled=False,\n",
")\n",
"\n",
"model_to_run"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6ddd57de-9f41-403c-bccc-8d3118654a24",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "af1d662119844324a207cc5f47dfd126",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Dropdown(description='Device:', options=('CPU', 'GPU.0', 'GPU.1', 'AUTO'), value='CPU')"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import ipywidgets as widgets\n",
"import openvino as ov\n",
"\n",
"core = ov.Core()\n",
"\n",
"device = widgets.Dropdown(\n",
" options=core.available_devices + [\"AUTO\"],\n",
" value=\"CPU\",\n",
" description=\"Device:\",\n",
" disabled=False,\n",
")\n",
"\n",
"device"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "dbea489f-7ff1-49a1-a14d-5d19fd0abfb2",
"metadata": {},
"source": [
"## Instantiate Model using Optimum Intel\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"Optimum Intel can be used to load optimized models from the [Hugging Face Hub](https://huggingface.co/docs/optimum/intel/hf.co/models) and create pipelines to run an inference with OpenVINO Runtime using Hugging Face APIs. The Optimum Inference models are API compatible with Hugging Face Transformers models. This means we just need to replace `AutoModelForXxx` class with the corresponding `OVModelForXxx` class.\n",
"\n",
"Below is an example of the Dolly model\n",
"\n",
"```diff\n",
"-from transformers import AutoModelForCausalLM\n",
"+from optimum.intel.openvino import OVModelForCausalLM\n",
"from transformers import AutoTokenizer, pipeline\n",
"\n",
"model_id = \"databricks/dolly-v2-3b\"\n",
"-model = AutoModelForCausalLM.from_pretrained(model_id)\n",
"+model = OVModelForCausalLM.from_pretrained(model_id, export=True)\n",
"```\n",
"\n",
"Model class initialization starts with calling `from_pretrained` method. When downloading and converting Transformers model, the parameter `export=True` should be added (as we already converted model before, we do not need to provide this parameter). We can save the converted model for the next usage with the `save_pretrained` method.\n",
"Tokenizer class and pipelines API are compatible with Optimum models.\n",
"\n",
"You can find more details about OpenVINO LLM inference using HuggingFace Optimum API in [LLM inference guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "91f42296-627d-44ff-a1cb-936bb6f87992",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, tensorflow, onnx, openvino\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'\n",
"2024-05-01 10:43:29.010748: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
"2024-05-01 10:43:29.012724: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
"2024-05-01 10:43:29.047558: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
"2024-05-01 10:43:29.048434: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
"To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"2024-05-01 10:43:29.742257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
"/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.\n",
" warn(\"The installed version of bitsandbytes was compiled without GPU support. \"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/ea/work/my_optimum_intel/optimum_env/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:\n",
" PyTorch 2.0.1+cu118 with CUDA 1108 (you have 2.1.2+cpu)\n",
" Python 3.8.18 (you have 3.8.10)\n",
" Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)\n",
" Memory-efficient attention, SwiGLU, sparse and more won't be available.\n",
" Set XFORMERS_MORE_DETAILS=1 for more details\n",
"Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading model from dolly-v2-3b/INT4_compressed_weights\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Compiling the model to CPU ...\n"
]
}
],
"source": [
"from pathlib import Path\n",
"from transformers import AutoTokenizer\n",
"from optimum.intel.openvino import OVModelForCausalLM\n",
"\n",
"if model_to_run.value == \"INT4\":\n",
" model_dir = int4_model_dir\n",
"elif model_to_run.value == \"INT8\":\n",
" model_dir = int8_model_dir\n",
"else:\n",
" model_dir = fp16_model_dir\n",
"print(f\"Loading model from {model_dir}\")\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(model_dir)\n",
"\n",
"current_device = device.value\n",
"\n",
"ov_config = {\"PERFORMANCE_HINT\": \"LATENCY\", \"NUM_STREAMS\": \"1\", \"CACHE_DIR\": \"\"}\n",
"\n",
"ov_model = OVModelForCausalLM.from_pretrained(model_dir, device=current_device, ov_config=ov_config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b6d9c4a5-ef75-4076-9f1c-f45a2259ec46",
"metadata": {},
"source": [
"## Create an instruction-following inference pipeline\n",
"[back to top ⬆️](#Table-of-contents:)\n",
" \n",
" The `run_generation` function accepts user-provided text input, tokenizes it, and runs the generation process. Text generation is an iterative process, where each next token depends on previously generated until a maximum number of tokens or stop generation condition is not reached. To obtain intermediate generation results without waiting until when generation is finished, we will use [`TextIteratorStreamer`](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.TextIteratorStreamer), provided as part of HuggingFace [Streaming API](https://huggingface.co/docs/transformers/main/en/generation_strategies#streaming).\n",
" \n",
"The diagram below illustrates how the instruction-following pipeline works\n",
"\n",
"\n",
"\n",
"As can be seen, on the first iteration, the user provided instructions converted to token ids using a tokenizer, then prepared input provided to the model. The model generates probabilities for all tokens in logits format The way the next token will be selected over predicted probabilities is driven by the selected decoding methodology. You can find more information about the most popular decoding methods in this [blog](https://huggingface.co/blog/how-to-generate).\n",
"\n",
"There are several parameters that can control text generation quality:\n",
"\n",
" * `Temperature` is a parameter used to control the level of creativity in AI-generated text. By adjusting the `temperature`, you can influence the AI model's probability distribution, making the text more focused or diverse. \n",
" Consider the following example: The AI model has to complete the sentence \"The cat is ____.\" with the following token probabilities: \n",
"\n",
" playing: 0.5 \n",
" sleeping: 0.25 \n",
" eating: 0.15 \n",
" driving: 0.05 \n",
" flying: 0.05 \n",
"\n",
" - **Low temperature** (e.g., 0.2): The AI model becomes more focused and deterministic, choosing tokens with the highest probability, such as \"playing.\" \n",
" - **Medium temperature** (e.g., 1.0): The AI model maintains a balance between creativity and focus, selecting tokens based on their probabilities without significant bias, such as \"playing,\" \"sleeping,\" or \"eating.\" \n",
" - **High temperature** (e.g., 2.0): The AI model becomes more adventurous, increasing the chances of selecting less likely tokens, such as \"driving\" and \"flying.\"\n",
" * `Top-p`, also known as nucleus sampling, is a parameter used to control the range of tokens considered by the AI model based on their cumulative probability. By adjusting the `top-p` value, you can influence the AI model's token selection, making it more focused or diverse.\n",
" Using the same example with the cat, consider the following top_p settings: \n",
" - **Low top_p** (e.g., 0.5): The AI model considers only tokens with the highest cumulative probability, such as \"playing.\" \n",
" - **Medium top_p** (e.g., 0.8): The AI model considers tokens with a higher cumulative probability, such as \"playing,\" \"sleeping,\" and \"eating.\" \n",
" - **High top_p** (e.g., 1.0): The AI model considers all tokens, including those with lower probabilities, such as \"driving\" and \"flying.\" \n",
" * `Top-k` is another popular sampling strategy. In comparison with Top-P, which chooses from the smallest possible set of words whose cumulative probability exceeds the probability P, in Top-K sampling K most likely next words are filtered and the probability mass is redistributed among only those K next words. In our example with cat, if k=3, then only \"playing\", \"sleeping\" and \"eating\" will be taken into account as possible next word.\n",
"\n",
"To optimize the generation process and use memory more efficiently, the `use_cache=True` option is enabled. Since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step. Therefore, recomputing it every time you want to generate a new token seems wasteful. With the cache, the model saves the hidden state once it has been computed. The model only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from O(n^3) to O(n^2) for a transformer model. More details about how it works can be found in this [article](https://scale.com/blog/pytorch-improvements#Text%20Translation). With this option, the model gets the previous step's hidden states (cached attention keys and values) as input and additionally provides hidden states for the current step as output. It means for all next iterations, it is enough to provide only a new token obtained from the previous step and cached key values to get the next token prediction. \n",
"\n",
"The generation cycle repeats until the end of the sequence token is reached or it also can be interrupted when maximum tokens will be generated. As already mentioned before, we can enable printing current generated tokens without waiting until when the whole generation is finished using Streaming API, it adds a new token to the output queue and then prints them when they are ready."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b9b5da4d-d2fd-440b-b204-7fbc6966dd1f",
"metadata": {},
"source": [
"### Setup imports\n",
"[back to top ⬆️](#Table-of-contents:)\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6f976094-8603-42c4-8f18-a32ba6d7192e",
"metadata": {},
"outputs": [],
"source": [
"from threading import Thread\n",
"from time import perf_counter\n",
"from typing import List\n",
"import gradio as gr\n",
"from transformers import AutoTokenizer, TextIteratorStreamer\n",
"import numpy as np"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c58611d6-0a91-4efd-976e-4221acbb43cd",
"metadata": {},
"source": [
"### Prepare template for user prompt\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"For effective generation, model expects to have input in specific format. The code below prepare template for passing user instruction into model with providing additional context."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "52ac10a5-3141-4227-8f0b-0617acd027c8",
"metadata": {},
"outputs": [],
"source": [
"INSTRUCTION_KEY = \"### Instruction:\"\n",
"RESPONSE_KEY = \"### Response:\"\n",
"END_KEY = \"### End\"\n",
"INTRO_BLURB = \"Below is an instruction that describes a task. Write a response that appropriately completes the request.\"\n",
"\n",
"# This is the prompt that is used for generating responses using an already trained model. It ends with the response\n",
"# key, where the job of the model is to provide the completion that follows it (i.e. the response itself).\n",
"PROMPT_FOR_GENERATION_FORMAT = \"\"\"{intro}\n",
"\n",
"{instruction_key}\n",
"{instruction}\n",
"\n",
"{response_key}\n",
"\"\"\".format(\n",
" intro=INTRO_BLURB,\n",
" instruction_key=INSTRUCTION_KEY,\n",
" instruction=\"{instruction}\",\n",
" response_key=RESPONSE_KEY,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "27a01739-1363-42ef-927f-6a340bdbe7ba",
"metadata": {},
"source": [
"### Helpers for output parsing\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"Model was retrained to finish generation using special token `### End` the code below find its id for using it as generation stop-criteria."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "524e72f4-8750-48ff-b002-e558d03b3302",
"metadata": {},
"outputs": [],
"source": [
"def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:\n",
" \"\"\"\n",
" Gets the token ID for a given string that has been added to the tokenizer as a special token.\n",
"\n",
" When training, we configure the tokenizer so that the sequences like \"### Instruction:\" and \"### End\" are\n",
" treated specially and converted to a single, new token. This retrieves the token ID each of these keys map to.\n",
"\n",
" Args:\n",
" tokenizer (PreTrainedTokenizer): the tokenizer\n",
" key (str): the key to convert to a single token\n",
"\n",
" Raises:\n",
" RuntimeError: if more than one ID was generated\n",
"\n",
" Returns:\n",
" int: the token ID for the given key\n",
" \"\"\"\n",
" token_ids = tokenizer.encode(key)\n",
" if len(token_ids) > 1:\n",
" raise ValueError(f\"Expected only a single token for '{key}' but found {token_ids}\")\n",
" return token_ids[0]\n",
"\n",
"\n",
"tokenizer_response_key = next(\n",
" (token for token in tokenizer.additional_special_tokens if token.startswith(RESPONSE_KEY)),\n",
" None,\n",
")\n",
"\n",
"end_key_token_id = None\n",
"if tokenizer_response_key:\n",
" try:\n",
" end_key_token_id = get_special_token_id(tokenizer, END_KEY)\n",
" # Ensure generation stops once it generates \"### End\"\n",
" except ValueError:\n",
" pass"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "583202d2-6d29-4729-af2e-232d3ee0bc2c",
"metadata": {},
"source": [
"### Main generation function\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"As it was discussed above, `run_generation` function is the entry point for starting generation. It gets provided input instruction as parameter and returns model response."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "67fb4f9d-5877-48d8-8eff-c30ff6974d7a",
"metadata": {},
"outputs": [],
"source": [
"def run_generation(\n",
" user_text: str,\n",
" top_p: float,\n",
" temperature: float,\n",
" top_k: int,\n",
" max_new_tokens: int,\n",
" perf_text: str,\n",
"):\n",
" \"\"\"\n",
" Text generation function\n",
"\n",
" Parameters:\n",
" user_text (str): User-provided instruction for a generation.\n",
" top_p (float): Nucleus sampling. If set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
" temperature (float): The value used to module the logits distribution.\n",
" top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
" max_new_tokens (int): Maximum length of generated sequence.\n",
" perf_text (str): Content of text field for printing performance results.\n",
" Returns:\n",
" model_output (str) - model-generated text\n",
" perf_text (str) - updated perf text filed content\n",
" \"\"\"\n",
"\n",
" # Prepare input prompt according to model expected template\n",
" prompt_text = PROMPT_FOR_GENERATION_FORMAT.format(instruction=user_text)\n",
"\n",
" # Tokenize the user text.\n",
" model_inputs = tokenizer(prompt_text, return_tensors=\"pt\")\n",
"\n",
" # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer\n",
" # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.\n",
" streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n",
" generate_kwargs = dict(\n",
" model_inputs,\n",
" streamer=streamer,\n",
" max_new_tokens=max_new_tokens,\n",
" do_sample=True,\n",
" top_p=top_p,\n",
" temperature=float(temperature),\n",
" top_k=top_k,\n",
" eos_token_id=end_key_token_id,\n",
" )\n",
" t = Thread(target=ov_model.generate, kwargs=generate_kwargs)\n",
" t.start()\n",
"\n",
" # Pull the generated text from the streamer, and update the model output.\n",
" model_output = \"\"\n",
" per_token_time = []\n",
" num_tokens = 0\n",
" start = perf_counter()\n",
" for new_text in streamer:\n",
" current_time = perf_counter() - start\n",
" model_output += new_text\n",
" perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)\n",
" yield model_output, perf_text\n",
" start = perf_counter()\n",
" return model_output, perf_text"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "562f2dcf-75ef-4554-85e3-e04f486776cc",
"metadata": {},
"source": [
"### Helpers for application\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"For making interactive user interface we will use Gradio library. The code bellow provides useful functions used for communication with UI elements."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "f114944f-c060-44ba-ba59-02cb2516554c",
"metadata": {},
"outputs": [],
"source": [
"def estimate_latency(\n",
" current_time: float,\n",
" current_perf_text: str,\n",
" new_gen_text: str,\n",
" per_token_time: List[float],\n",
" num_tokens: int,\n",
"):\n",
" \"\"\"\n",
" Helper function for performance estimation\n",
"\n",
" Parameters:\n",
" current_time (float): This step time in seconds.\n",
" current_perf_text (str): Current content of performance UI field.\n",
" new_gen_text (str): New generated text.\n",
" per_token_time (List[float]): history of performance from previous steps.\n",
" num_tokens (int): Total number of generated tokens.\n",
"\n",
" Returns:\n",
" update for performance text field\n",
" update for a total number of tokens\n",
" \"\"\"\n",
" num_current_toks = len(tokenizer.encode(new_gen_text))\n",
" num_tokens += num_current_toks\n",
" per_token_time.append(num_current_toks / current_time)\n",
" if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:\n",
" current_bucket = per_token_time[:-10]\n",
" return (\n",
" f\"Average generation speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}\",\n",
" num_tokens,\n",
" )\n",
" return current_perf_text, num_tokens\n",
"\n",
"\n",
"def reset_textbox(instruction: str, response: str, perf: str):\n",
" \"\"\"\n",
" Helper function for resetting content of all text fields\n",
"\n",
" Parameters:\n",
" instruction (str): Content of user instruction field.\n",
" response (str): Content of model response field.\n",
" perf (str): Content of performance info filed\n",
"\n",
" Returns:\n",
" empty string for each placeholder\n",
" \"\"\"\n",
" return \"\", \"\", \"\"\n",
"\n",
"\n",
"def select_device(device_str: str, current_text: str = \"\", progress: gr.Progress = gr.Progress()):\n",
" \"\"\"\n",
" Helper function for uploading model on the device.\n",
"\n",
" Parameters:\n",
" device_str (str): Device name.\n",
" current_text (str): Current content of user instruction field (used only for backup purposes, temporally replacing it on the progress bar during model loading).\n",
" progress (gr.Progress): gradio progress tracker\n",
" Returns:\n",
" current_text\n",
" \"\"\"\n",
" if device_str != ov_model._device:\n",
" ov_model.request = None\n",
" ov_model._device = device_str\n",
"\n",
" for i in progress.tqdm(range(1), desc=f\"Model loading on {device_str}\"):\n",
" ov_model.compile()\n",
" return current_text"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "50d918a9-1cbe-49a5-85ad-5e370c8af7f5",
"metadata": {},
"source": [
"## Run instruction-following pipeline\n",
"[back to top ⬆️](#Table-of-contents:)\n",
"\n",
"Now, we are ready to explore model capabilities. This demo provides a simple interface that allows communication with a model using text instruction. Type your instruction into the `User instruction` field or select one from predefined examples and click on the `Submit` button to start generation. Additionally, you can modify advanced generation parameters:\n",
"\n",
"* `Device` - allows switching inference device. Please note, every time when new device is selected, model will be recompiled and this takes some time.\n",
"* `Max New Tokens` - maximum size of generated text.\n",
"* `Top-p (nucleus sampling)` - if set to < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.\n",
"* `Top-k` - the number of highest probability vocabulary tokens to keep for top-k-filtering.\n",
"* `Temperature` - the value used to module the logits distribution."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "a00c2293-15b1-4734-b9b4-1abb524bb8d6",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"available_devices = ov.Core().available_devices + [\"AUTO\"]\n",
"\n",
"examples = [\n",
" \"Give me recipe for pizza with pineapple\",\n",
" \"Write me a tweet about new OpenVINO release\",\n",
" \"Explain difference between CPU and GPU\",\n",
" \"Give five ideas for great weekend with family\",\n",
" \"Do Androids dream of Electric sheep?\",\n",
" \"Who is Dolly?\",\n",
" \"Please give me advice how to write resume?\",\n",
" \"Name 3 advantages to be a cat\",\n",
" \"Write instructions on how to become a good AI engineer\",\n",
" \"Write a love letter to my best friend\",\n",
"]\n",
"\n",
"with gr.Blocks() as demo:\n",
" gr.Markdown(\n",
" \"# Instruction following using Databricks Dolly 2.0 and OpenVINO.\\n\"\n",
" \"Provide insturction which describes a task below or select among predefined examples and model writes response that performs requested task.\"\n",
" )\n",
"\n",
" with gr.Row():\n",
" with gr.Column(scale=4):\n",
" user_text = gr.Textbox(\n",
" placeholder=\"Write an email about an alpaca that likes flan\",\n",
" label=\"User instruction\",\n",
" )\n",
" model_output = gr.Textbox(label=\"Model response\", interactive=False)\n",
" performance = gr.Textbox(label=\"Performance\", lines=1, interactive=False)\n",
" with gr.Column(scale=1):\n",
" button_clear = gr.Button(value=\"Clear\")\n",
" button_submit = gr.Button(value=\"Submit\")\n",
" gr.Examples(examples, user_text)\n",
" with gr.Column(scale=1):\n",
" device = gr.Dropdown(choices=available_devices, value=current_device, label=\"Device\")\n",
" max_new_tokens = gr.Slider(\n",
" minimum=1,\n",
" maximum=1000,\n",
" value=256,\n",
" step=1,\n",
" interactive=True,\n",
" label=\"Max New Tokens\",\n",
" )\n",
" top_p = gr.Slider(\n",
" minimum=0.05,\n",
" maximum=1.0,\n",
" value=0.92,\n",
" step=0.05,\n",
" interactive=True,\n",
" label=\"Top-p (nucleus sampling)\",\n",
" )\n",
" top_k = gr.Slider(\n",
" minimum=0,\n",
" maximum=50,\n",
" value=0,\n",
" step=1,\n",
" interactive=True,\n",
" label=\"Top-k\",\n",
" )\n",
" temperature = gr.Slider(\n",
" minimum=0.1,\n",
" maximum=5.0,\n",
" value=0.8,\n",
" step=0.1,\n",
" interactive=True,\n",
" label=\"Temperature\",\n",
" )\n",
"\n",
" user_text.submit(\n",
" run_generation,\n",
" [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
" [model_output, performance],\n",
" )\n",
" button_submit.click(select_device, [device, user_text], [user_text])\n",
" button_submit.click(\n",
" run_generation,\n",
" [user_text, top_p, temperature, top_k, max_new_tokens, performance],\n",
" [model_output, performance],\n",
" )\n",
" button_clear.click(\n",
" reset_textbox,\n",
" [user_text, model_output, performance],\n",
" [user_text, model_output, performance],\n",
" )\n",
" device.change(select_device, [device, user_text], [user_text])\n",
"\n",
"if __name__ == \"__main__\":\n",
" try:\n",
" demo.queue().launch(debug=True, height=800)\n",
" except Exception:\n",
" demo.queue().launch(debug=True, share=True, height=800)\n",
"\n",
"# If you are launching remotely, specify server_name and server_port\n",
"# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
"# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"openvino_notebooks": {
"imageUrl": "https://user-images.githubusercontent.com/29454499/237291423-022f07d2-966b-4be2-9a1c-98f1cf0691c2.png",
"tags": {
"categories": [
"Model Demos",
"AI Trends"
],
"libraries": [],
"other": [
"LLM"
],
"tasks": [
"Text Generation"
]
}
},
"vscode": {
"interpreter": {
"hash": "cec18e25feb9469b5ff1085a8097bdcd86db6a4ac301d6aeff87d0f3e7ce4ca5"
}
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|