File size: 53,943 Bytes
76d705d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 |
{
"cells": [
{
"cell_type": "markdown",
"id": "90c6730f-5d76-450b-9788-ec883d024f57",
"metadata": {},
"source": [
"# Hugging Face Transformers 微调训练入门\n",
"\n",
"本示例将介绍基于 Transformers 实现模型微调训练的主要流程，包括：\n",
"- 数据集下载\n",
"- 数据预处理\n",
"- 训练超参数配置\n",
"- 训练评估指标设置\n",
"- 训练器基本介绍\n",
"- 实战训练\n",
"- 模型保存"
]
},
{
"cell_type": "markdown",
"id": "aa0b1e12-1921-4438-8d5d-9760a629dcfe",
"metadata": {},
"source": [
"## YelpReviewFull 数据集\n",
"\n",
"**Hugging Face 数据集：[ YelpReviewFull ](https://huggingface.co/datasets/yelp_review_full)**\n",
"\n",
"### 数据集摘要\n",
"\n",
"Yelp评论数据集包括来自Yelp的评论。它是从Yelp Dataset Challenge 2015数据中提取的。\n",
"\n",
"### 支持的任务和排行榜\n",
"文本分类、情感分类：该数据集主要用于文本分类：给定文本，预测情感。\n",
"\n",
"### 语言\n",
"这些评论主要以英语编写。\n",
"\n",
"### 数据集结构\n",
"\n",
"#### 数据实例\n",
"一个典型的数据点包括文本和相应的标签。\n",
"\n",
"来自YelpReviewFull测试集的示例如下：\n",
"\n",
"```json\n",
"{\n",
" 'label': 0,\n",
" 'text': 'I got \\'new\\' tires from them and within two weeks got a flat. I took my car to a local mechanic to see if i could get the hole patched, but they said the reason I had a flat was because the previous patch had blown - WAIT, WHAT? I just got the tire and never needed to have it patched? This was supposed to be a new tire. \\\\nI took the tire over to Flynn\\'s and they told me that someone punctured my tire, then tried to patch it. So there are resentful tire slashers? I find that very unlikely. After arguing with the guy and telling him that his logic was far fetched he said he\\'d give me a new tire \\\\\"this time\\\\\". \\\\nI will never go back to Flynn\\'s b/c of the way this guy treated me and the simple fact that they gave me a used tire!'\n",
"}\n",
"```\n",
"\n",
"#### 数据字段\n",
"\n",
"- 'text': 评论文本使用双引号（\"）转义，任何内部双引号都通过2个双引号（\"\"）转义。换行符使用反斜杠后跟一个 \"n\" 字符转义，即 \"\\n\"。\n",
"- 'label': 对应于评论的分数（介于1和5之间）。\n",
"\n",
"#### 数据拆分\n",
"\n",
"Yelp评论完整星级数据集是通过随机选取每个1到5星评论的130,000个训练样本和10,000个测试样本构建的。总共有650,000个训练样本和50,000个测试样本。\n",
"\n",
"## 下载数据集"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9aa8ae5e-f57b-40cb-b929-16b172eed9a2",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"3"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import os\n",
"os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\" # see issue #152\n",
"os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"1,2,3\"\n",
"\n",
"import torch\n",
"torch.cuda.device_count()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "bbf72d6c-7ea5-4ee1-969a-c5060b9cb2d4",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.9/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"from datasets import load_dataset\n",
"\n",
"dataset = load_dataset(\"yelp_review_full\")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ec6fc806-1395-42dd-8121-a6e98a95cf01",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['label', 'text'],\n",
" num_rows: 650000\n",
" })\n",
" test: Dataset({\n",
" features: ['label', 'text'],\n",
" num_rows: 50000\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c94ad529-1604-48bd-8c8d-aa2f3bca6200",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'label': 0,\n",
" 'text': \"Owning a driving range inside the city limits is like a license to print money. I don't think I ask much out of a driving range. Decent mats, clean balls and accessible hours. Hell you need even less people now with the advent of the machine that doles out the balls. This place has none of them. It is april and there are no grass tees yet. BTW they opened for the season this week although it has been golfing weather for a month. The mats look like the carpet at my 107 year old aunt Irene's house. Worn and thread bare. Let's talk about the hours. This place is equipped with lights yet they only sell buckets of balls until 730. It is still light out. Finally lets you have the pit to hit into. When I arrived I wasn't sure if this was a driving range or an excavation site for a mastodon or a strip mining operation. There is no grass on the range. Just mud. Makes it a good tool to figure out how far you actually are hitting the ball. Oh, they are cash only also.\\\\n\\\\nBottom line, this place sucks. The best hope is that the owner sells it to someone that actually wants to make money and service golfers in Pittsburgh.\"}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[\"train\"][10]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "6dc45997-e391-456f-b0b9-d3193b0f6a9d",
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import pandas as pd\n",
"import datasets\n",
"from IPython.display import display, HTML"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9e2ecebb-d5d1-456d-967c-842a79fdd622",
"metadata": {},
"outputs": [],
"source": [
"def show_random_elements(dataset, num_examples=10):\n",
" assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
" picks = []\n",
" for _ in range(num_examples):\n",
" pick = random.randint(0, len(dataset)-1)\n",
" while pick in picks:\n",
" pick = random.randint(0, len(dataset)-1)\n",
" picks.append(pick)\n",
" \n",
" df = pd.DataFrame(dataset[picks])\n",
" for column, typ in dataset.features.items():\n",
" if isinstance(typ, datasets.ClassLabel):\n",
" df[column] = df[column].transform(lambda i: typ.names[i])\n",
" display(HTML(df.to_html()))"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1af560b6-7d21-499e-9b82-114be371a98a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>label</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4 stars</td>\n",
" <td>Great pizza and Sweet Chili wings in The Brewer's Cafe. My wife and I ordered their pizza. I have tried MANY pizzas. My wife and I were pleasantly surprised! My only complaints about this place would be that the menu is just a tad too small. Another tip: Anything fried comes out PIPING HOT!!! ;-)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2 star</td>\n",
" <td>This is the exact reason why Yelp at times is not a credible resource. You see legitimate users with legitimate reviews frequently get filtered, and you see 4-5 star ratings for restaurants that provide a low quality product. \\n\\nFor years, I kept reading on Yelp that \\\"Blueberry Hill\\\" has the \\\"greatest hand breaded chicken fried steak in all of Las Vegas\\\". There was nothing true about that statement. The chicken fried steak was nothing more than the processed precooked version that you get at Costco and Smart N Final. All they do is reheat it and serve. I was very disappointed on account of all the good reviews on yelp about how the steak was a \\\"real sirloin that was hand battered\\\". \\n\\nThe side salad was merely lettuce with 1 ring of red onion, served in a beaten up plastic bowl. No tomatoes, no cheese, no croutons...nothing. The Ranch dressing tasted like water mixed with mayonnaise. \\n\\nThe biscuit that came with our steaks tasted like garlic bread. \\n\\nThe corn bread that came with our steak never came out. The server forgot to bring it out. We never got it. \\n\\nOn a positive note. The Buser was very hard working and made up for the poor quality service that the server provided. The server was not rude, but she was very forgetful and never came around one time to check on us. You need to flag down the server to get her attention. The Buser compensated for the lack of service by coming around to refill our coffee and water. The server was usually found sitting at the counter and reading a book.\\n\\nWhen I asked the server for Tabasco, she forgot. 10 minutes later I asked her again and she finally got it. When I asked the server for Jam, she forgot. When I asked her for saltine crackers, she forgot. She did bring out the check immediately though before we even touched our food. \\n\\nIn conclusion. The bill came out quicker than the food. The buser was the hardest working guy here. The food was processed and low quality. The prices were pretty high. Yelp has failed us all.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2 star</td>\n",
" <td>For the $18 ticket price, this aquarium was way smaller than I expected. I wish we had taken our time checking things out. We sped through each exhibit thinking there would be more impressive things ahead, but we reached the end quickly and thought, \\\"Is that it?\\\" My nieces had a blast, but they are 4 and 3 and were also amazed by the monorail.\\n\\nThe jellyfish tank and the stingray pool were cool, but nothing stood out to make this place a must-do. One of the employees was more interested in talking about the Lakers instead of the piranhas.\\n\\nIf you're in the area and you need to entertain some kids, it might be ok. If not, skip it and hit the pool instead.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>4 stars</td>\n",
" <td>This was a last minute, drive by decision that fit our timing on a late weekday afternoon. What a wonderful surprise! Our timing must have been good, cuz we got right in with no wait.\\n\\nWe got in on the All You Can Eat sushi lunch for 20 bucks. The rolls were really good and the staff was super attentive. \\n\\nTiger roll, Mount Charelston roll - thumbs up. The spicy scallop? Seems to be a different standard out west, and not my favorite Tsu Kasa style. The tuna was sublime. Seaweed salad - the perfect serving size. Shrimp Tempura - amazingly perfect. Tried the Ikura (Salmon Roe) but realized I hadn't quite evolved that far.\\n\\nI would go back to this place again and again.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>5 stars</td>\n",
" <td>Not sure why people are writing bad reviews over things out of the clubs control. Going over a few things \\n\\n1.) the most impressive part of this club is the DJs. I've been here twice and both times every dj playing was spot on. \\n\\n2) I personally like the dress code. An upscale gay bar on the strip is much needed. \\n\\n3) I do wish it wasn't in ballys, it's a bit of an annoyance being right in the middle of the chaos of the strip but overall was easy to get to. \\n\\n4) best advice is to try and get on the guest list. Drinks aren't cheap but this is vegas. Find me a cocktail in a nightclub on the strip under 10 bucks and we can say this place is expensive \\n\\n\\nKeep up the great work!</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>5 stars</td>\n",
" <td>Love the juice bar! Try the kaleaid. They now have bottled cold pressed juice. It's expensive, but so good.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>3 stars</td>\n",
" <td>As someone who doesn't drink I am left to base my review on the ambiance and food. Despite being in a South Scottsdale strip mall the crowd is fun and engaging. As has been mentioned by the majority of the other reviews, the food isn't really the draw of this place. Still a decent way to pass an evening with friends watching games.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>4 stars</td>\n",
" <td>One of my favorite places for Indian food. Their lunch buffet selection is good and all of their food is delicious! Staff is very friendly as well.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>4 stars</td>\n",
" <td>I loved my sandwich. The Bobbie tastes like Thanksgiving! It's turkey, w/ stuffing, and cranberry on fresh bread. For reals! Extra yum! ...The two dudes that made our sandwiches were cool.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>1 star</td>\n",
" <td>Do Not Go here They Just Butcherd my Cuticles the guy didn't speak any English and the owners wife was yelling at me because I was upset with my service.. Then she locked me in the place so I called the police on her and her husband.. They Tryed to Chase Me And Then Locked Me inside.. And I even payed her in full and she didn't even finish her job Because she was yelling so loud in the place.. DO NOT WASTE YOUR TIME AND MONEY..LESSON LEARNED NEVER TO GO TO PLACES LIKE THIS EVER AGAIN..</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_random_elements(dataset[\"train\"])"
]
},
{
"cell_type": "markdown",
"id": "c9df7cd0-23cd-458f-b2b5-f025c3b9fe62",
"metadata": {},
"source": [
"## 预处理数据\n",
"\n",
"下载数据集到本地后，使用 Tokenizer 来处理文本，对于长度不等的输入数据，可以使用填充（padding）和截断（truncation）策略来处理。\n",
"\n",
"Datasets 的 `map` 方法，支持一次性在整个数据集上应用预处理函数。\n",
"\n",
"下面使用填充到最大长度的策略，处理整个数据集："
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "8bf2b342-e1dd-4ab6-ad57-28eb2513ae38",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
"\n",
"\n",
"def tokenize_function(examples):\n",
" return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
"\n",
"\n",
"tokenized_datasets = dataset.map(tokenize_function, batched=True)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "47a415a8-cd15-4a8c-851b-9b4740ef8271",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>label</th>\n",
" <th>text</th>\n",
" <th>input_ids</th>\n",
" <th>token_type_ids</th>\n",
" <th>attention_mask</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3 stars</td>\n",
" <td>Visited Hillstone to give them another try a couple weeks ago. Finally had the time to sit down and write an update.\\n\\nAs usual the food was great. The service, while better this occasion was still a bit off. Two examples:\\n\\n1) Length of time for wait staff to revisit our table for refreshments was fairly lengthy in comparison to the old Houston's.\\n\\n2) When asked about the sauce used in the Thai Tuna Roll, the waitress looked very blank and then said it was in the roll. She replied, tuna and some sort of Thai sauce. Noooo kidding.\\n\\nI will give this place another chance and another star because I really like their food. I will more than likely return. However, I will update with new reviews until it's back to the place I remember.</td>\n",
" <td>[101, 159, 26868, 1906, 5377, 4793, 1106, 1660, 1172, 1330, 2222, 170, 2337, 2277, 2403, 119, 4428, 1125, 1103, 1159, 1106, 3465, 1205, 1105, 3593, 1126, 11984, 119, 165, 183, 165, 183, 23390, 4400, 1103, 2094, 1108, 1632, 119, 1109, 1555, 117, 1229, 1618, 1142, 6116, 1108, 1253, 170, 2113, 1228, 119, 1960, 5136, 131, 165, 183, 165, 183, 1475, 114, 16758, 1104, 1159, 1111, 3074, 2546, 1106, 1231, 9356, 2875, 1412, 1952, 1111, 1231, 2087, 21298, 4385, 1108, 6751, 12628, 1107, 7577, 1106, 1103, 1385, 4666, 112, 188, 119, 165, 183, 165, 183, 1477, 114, 1332, 1455, 1164, 1103, ...]</td>\n",
" <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]</td>\n",
" <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...]</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show_random_elements(tokenized_datasets[\"train\"], num_examples=1)"
]
},
{
"cell_type": "markdown",
"id": "1c33d153-f729-4f04-972c-a764c1cbbb8b",
"metadata": {},
"source": [
"### 数据抽样\n",
"\n",
"使用 1000 个数据样本，在 BERT 上演示小规模训练（基于 Pytorch Trainer）\n",
"\n",
"`shuffle()`函数会随机重新排列列的值。如果您希望对用于洗牌数据集的算法有更多控制，可以在此函数中指定generator参数来使用不同的numpy.random.Generator。"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "a17317d8-3c6a-467f-843d-87491f600db1",
"metadata": {},
"outputs": [],
"source": [
"# small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
"# small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))"
]
},
{
"cell_type": "markdown",
"id": "d3b65d63-2d3a-4a56-bc31-6e88a29e9dec",
"metadata": {},
"source": [
"## 微调训练配置\n",
"\n",
"### 加载 BERT 模型\n",
"\n",
"警告通知我们正在丢弃一些权重（`vocab_transform` 和 `vocab_layer_norm` 层），并随机初始化其他一些权重（`pre_classifier` 和 `classifier` 层）。在微调模型情况下是绝对正常的，因为我们正在删除用于预训练模型的掩码语言建模任务的头部，并用一个新的头部替换它，对于这个新头部，我们没有预训练的权重，所以库会警告我们在用它进行推理之前应该对这个模型进行微调，而这正是我们要做的事情。"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "4d2af4df-abd4-4a4b-94b6-b0e7375304ed",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"from transformers import AutoModelForSequenceClassification\n",
"\n",
"model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=5)"
]
},
{
"cell_type": "markdown",
"id": "b44014df-b52c-4c72-9e9f-54424725a473",
"metadata": {},
"source": [
"### 训练超参数（TrainingArguments）\n",
"\n",
"完整配置参数与默认值：https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments\n",
"\n",
"源代码定义：https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/training_args.py#L161\n",
"\n",
"**最重要配置：模型权重保存路径(output_dir)**"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "98c01d5c-de72-4ff0-b11d-e07ac5346888",
"metadata": {},
"outputs": [],
"source": [
"# from transformers import TrainingArguments\n",
"\n",
"# model_dir = \"models/bert-base-cased\"\n",
"\n",
"# # logging_steps 默认值为500，根据我们的训练数据和步长，将其设置为100\n",
"# training_args = TrainingArguments(output_dir=f\"{model_dir}/test_trainer\",\n",
"# logging_dir=f\"{model_dir}/test_trainer/runs\",\n",
"# logging_steps=100)\n",
"# # 完整的超参数配置\n",
"# print(training_args)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0ce03480-3aaa-48ea-a0c6-a177b8d8e34f",
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
}
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "7ebd3365-d359-4ab4-a300-4717590cc240",
"metadata": {},
"source": [
"### 训练过程中的指标评估（Evaluate)\n",
"\n",
"**[Hugging Face Evaluate 库](https://huggingface.co/docs/evaluate/index)** 支持使用一行代码，获得数十种不同领域（自然语言处理、计算机视觉、强化学习等）的评估方法。 当前支持 **完整评估指标：https://huggingface.co/evaluate-metric**\n",
"\n",
"训练器（Trainer）在训练过程中不会自动评估模型性能。因此，我们需要向训练器传递一个函数来计算和报告指标。 \n",
"\n",
"Evaluate库提供了一个简单的准确率函数，您可以使用`evaluate.load`函数加载"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "2a8ef138-5bf2-41e5-8c68-df8e11f4e98f",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import evaluate\n",
"\n",
"metric = evaluate.load(\"accuracy\")"
]
},
{
"cell_type": "markdown",
"id": "70d406c0-56d0-4a54-9c6c-e126ab7f5254",
"metadata": {},
"source": [
"\n",
"接着，调用 `compute` 函数来计算预测的准确率。\n",
"\n",
"在将预测传递给 compute 函数之前，我们需要将 logits 转换为预测值（**所有Transformers 模型都返回 logits**）。"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "f46d2e59-1ebf-43d2-bc86-6b57a4d24d19",
"metadata": {},
"outputs": [],
"source": [
"def compute_metrics(eval_pred):\n",
" logits, labels = eval_pred\n",
" predictions = np.argmax(logits, axis=-1)\n",
" return metric.compute(predictions=predictions, references=labels)"
]
},
{
"cell_type": "markdown",
"id": "e2feba67-9ca9-4793-9a15-3eaa426df2a1",
"metadata": {},
"source": [
"#### 训练过程指标监控\n",
"\n",
"通常，为了监控训练过程中的评估指标变化，我们可以在`TrainingArguments`指定`evaluation_strategy`参数，以便在 epoch 结束时报告评估指标。"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "afaaee18-4986-4e39-8ad9-b8d413ab4cd1",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"model_dir = \"models/bert-base-cased\"\n",
"batch_size = 14\n",
"\n",
"training_args = TrainingArguments(\n",
" output_dir=f\"{model_dir}/test_trainer\",\n",
" evaluation_strategy=\"epoch\", \n",
" logging_dir=f\"{model_dir}/test_trainer/runs\",\n",
" logging_steps=500,\n",
" save_total_limit=3,\n",
" per_device_train_batch_size=batch_size,\n",
" per_device_eval_batch_size=batch_size,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "d47d6981-e444-4c0f-a7cb-dd7f2ba8df12",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"## 开始训练\n",
"\n",
"### 实例化训练器（Trainer）\n",
"\n",
"`kernel version` 版本问题：暂不影响本示例代码运行"
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "ca1d12ac-89dc-4c30-8282-f859724c0062",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
"small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n",
"\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=small_eval_dataset,\n",
" compute_metrics=compute_metrics,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 64,
"id": "9b3c069d-a0dc-4f43-aea0-6cb8799643f3",
"metadata": {},
"outputs": [],
"source": [
"# trainer.args"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "449eb845-cff7-40ba-8915-38de79248840",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "a833e0db-1168-4a3c-8b75-bfdcef8c5157",
"metadata": {},
"source": [
"## 使用 nvidia-smi 查看 GPU 使用\n",
"\n",
"为了实时查看GPU使用情况，可以使用 `watch` 指令实现轮询：`watch -n 1 nvidia-smi`:\n",
"\n",
"```shell\n",
"Every 1.0s: nvidia-smi Wed Dec 20 14:37:41 2023\n",
"\n",
"Wed Dec 20 14:37:41 2023\n",
"+---------------------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |\n",
"|-----------------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|=========================================+======================+======================|\n",
"| 0 Tesla T4 Off | 00000000:00:0D.0 Off | 0 |\n",
"| N/A 64C P0 69W / 70W | 6665MiB / 15360MiB | 98% Default |\n",
"| | | N/A |\n",
"+-----------------------------------------+----------------------+----------------------+\n",
"\n",
"+---------------------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=======================================================================================|\n",
"| 0 N/A N/A 18395 C /root/miniconda3/bin/python 6660MiB |\n",
"+---------------------------------------------------------------------------------------+\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 65,
"id": "accfe921-471d-481a-96da-c491cdebad0c",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='46431' max='46431' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [46431/46431 17:33:04, Epoch 3/3]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>Accuracy</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.727000</td>\n",
" <td>0.694410</td>\n",
" <td>0.703000</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.633200</td>\n",
" <td>0.691635</td>\n",
" <td>0.710000</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.528600</td>\n",
" <td>0.732436</td>\n",
" <td>0.711000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"IOPub message rate exceeded.\n",
"The Jupyter server will temporarily stop sending output\n",
"to the client in order to avoid crashing it.\n",
"To change this limit, set the config variable\n",
"`--ServerApp.iopub_msg_rate_limit`.\n",
"\n",
"Current values:\n",
"ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
"ServerApp.rate_limit_window=3.0 (secs)\n",
"\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n",
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n"
]
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=46431, training_loss=0.6399251460484753, metrics={'train_runtime': 63185.9462, 'train_samples_per_second': 30.861, 'train_steps_per_second': 0.735, 'total_flos': 5.130803778048e+17, 'train_loss': 0.6399251460484753, 'epoch': 3.0})"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.train(False)"
]
},
{
"cell_type": "code",
"execution_count": 66,
"id": "6d581099-37a4-4470-b051-1ada38554089",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"small_test_dataset = tokenized_datasets[\"test\"].shuffle(seed=64).select(range(100))"
]
},
{
"cell_type": "code",
"execution_count": 67,
"id": "ffb47eab-1370-491e-8a84-6d5347a350b2",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
" warnings.warn('Was asked to gather along dimension 0, but all '\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='3' max='3' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [3/3 00:00]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.8645618557929993,\n",
" 'eval_accuracy': 0.65,\n",
" 'eval_runtime': 1.3182,\n",
" 'eval_samples_per_second': 75.861,\n",
" 'eval_steps_per_second': 2.276,\n",
" 'epoch': 3.0}"
]
},
"execution_count": 67,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.evaluate(small_test_dataset)"
]
},
{
"cell_type": "markdown",
"id": "27a55686-7c43-4ab8-a5cd-0e77f14c7c52",
"metadata": {},
"source": [
"### 保存模型和训练状态\n",
"\n",
"- 使用 `trainer.save_model` 方法保存模型，后续可以通过 from_pretrained() 方法重新加载\n",
"- 使用 `trainer.save_state` 方法保存训练状态"
]
},
{
"cell_type": "code",
"execution_count": 68,
"id": "ad0cbc14-9ef7-450f-a1a3-4f92b6486f41",
"metadata": {},
"outputs": [],
"source": [
"trainer.save_model(f\"{model_dir}/finetuned-trainer\")"
]
},
{
"cell_type": "code",
"execution_count": 69,
"id": "badf5868-2847-439d-a73e-42d1cca67b5e",
"metadata": {},
"outputs": [],
"source": [
"trainer.save_state()"
]
},
{
"cell_type": "markdown",
"id": "61828934-01da-4fc3-9e75-8d754c25dfbc",
"metadata": {},
"source": [
"## Homework: 使用完整的 YelpReviewFull 数据集训练，对比看 Acc 最高能到多少"
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "6ee2580a-7a5a-46ae-a28b-b41e9e838eb1",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"model.safetensors: 100%|██████████| 433M/433M [00:15<00:00, 28.0MB/s] \n"
]
},
{
"data": {
"text/plain": [
"CommitInfo(commit_url='https://huggingface.co/yqzhangjx/bert-base-cased-for-yelp/commit/ef8247a2eb2c3e93a70f0198591833256f6d197c', commit_message='Upload BertForSequenceClassification', commit_description='', oid='ef8247a2eb2c3e93a70f0198591833256f6d197c', pr_url=None, pr_revision=None, pr_num=None)"
]
},
"execution_count": 74,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.push_to_hub(\"yqzhangjx/bert-base-cased-for-yelp\", token=\"XXX\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "478f8d8e-2597-4a6c-a84c-d66e3d231e1d",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "561af3da-d720-4478-99de-b72d7419fb37",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|