Upload fine-tune-quickstart.ipynb

fine-tune-quickstart.ipynb

@@ -0,0 +1,1032 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "90c6730f-5d76-450b-9788-ec883d024f57",
+   "metadata": {},
+   "source": [
+    "# Hugging Face Transformers 微调训练入门\n",
+    "\n",
+    "本示例将介绍基于 Transformers 实现模型微调训练的主要流程，包括：\n",
+    "- 数据集下载\n",
+    "- 数据预处理\n",
+    "- 训练超参数配置\n",
+    "- 训练评估指标设置\n",
+    "- 训练器基本介绍\n",
+    "- 实战训练\n",
+    "- 模型保存"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aa0b1e12-1921-4438-8d5d-9760a629dcfe",
+   "metadata": {},
+   "source": [
+    "## YelpReviewFull 数据集\n",
+    "\n",
+    "**Hugging Face 数据集：[ YelpReviewFull ](https://huggingface.co/datasets/yelp_review_full)**\n",
+    "\n",
+    "### 数据集摘要\n",
+    "\n",
+    "Yelp评论数据集包括来自Yelp的评论。它是从Yelp Dataset Challenge 2015数据中提取的。\n",
+    "\n",
+    "### 支持的任务和排行榜\n",
+    "文本分类、情感分类：该数据集主要用于文本分类：给定文本，预测情感。\n",
+    "\n",
+    "### 语言\n",
+    "这些评论主要以英语编写。\n",
+    "\n",
+    "### 数据集结构\n",
+    "\n",
+    "#### 数据实例\n",
+    "一个典型的数据点包括文本和相应的标签。\n",
+    "\n",
+    "来自YelpReviewFull测试集的示例如下：\n",
+    "\n",
+    "```json\n",
+    "{\n",
+    "    'label': 0,\n",
+    "    'text': 'I got \\'new\\' tires from them and within two weeks got a flat. I took my car to a local mechanic to see if i could get the hole patched, but they said the reason I had a flat was because the previous patch had blown - WAIT, WHAT? I just got the tire and never needed to have it patched? This was supposed to be a new tire. \\\\nI took the tire over to Flynn\\'s and they told me that someone punctured my tire, then tried to patch it. So there are resentful tire slashers? I find that very unlikely. After arguing with the guy and telling him that his logic was far fetched he said he\\'d give me a new tire \\\\\"this time\\\\\". \\\\nI will never go back to Flynn\\'s b/c of the way this guy treated me and the simple fact that they gave me a used tire!'\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "#### 数据字段\n",
+    "\n",
+    "- 'text': 评论文本使用双引号（\"）转义，任何内部双引号都通过2个双引号（\"\"）转义。换行符使用反斜杠后跟一个 \"n\" 字符转义，即 \"\\n\"。\n",
+    "- 'label': 对应于评论的分数（介于1和5之间）。\n",
+    "\n",
+    "#### 数据拆分\n",
+    "\n",
+    "Yelp评论完整星级数据集是通过随机选取每个1到5星评论的130,000个训练样本和10,000个测试样本构建的。总共有650,000个训练样本和50,000个测试样本。\n",
+    "\n",
+    "## 下载数据集"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9aa8ae5e-f57b-40cb-b929-16b172eed9a2",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "3"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import os\n",
+    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\"   # see issue #152\n",
+    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"1,2,3\"\n",
+    "\n",
+    "import torch\n",
+    "torch.cuda.device_count()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "bbf72d6c-7ea5-4ee1-969a-c5060b9cb2d4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/lib/python3.9/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "from datasets import load_dataset\n",
+    "\n",
+    "dataset = load_dataset(\"yelp_review_full\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "ec6fc806-1395-42dd-8121-a6e98a95cf01",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "DatasetDict({\n",
+       "    train: Dataset({\n",
+       "        features: ['label', 'text'],\n",
+       "        num_rows: 650000\n",
+       "    })\n",
+       "    test: Dataset({\n",
+       "        features: ['label', 'text'],\n",
+       "        num_rows: 50000\n",
+       "    })\n",
+       "})"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c94ad529-1604-48bd-8c8d-aa2f3bca6200",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'label': 0,\n",
+       " 'text': \"Owning a driving range inside the city limits is like a license to print money.  I don't think I ask much out of a driving range.  Decent mats, clean balls and accessible hours.  Hell you need even less people now with the advent of the machine that doles out the balls.  This place has none of them.  It is april and there are no grass tees yet.  BTW they opened for the season this week although it has been golfing weather for a month.  The mats look like the carpet at my 107 year old aunt Irene's house.  Worn and thread bare.  Let's talk about the hours.  This place is equipped with lights yet they only sell buckets of balls until 730.  It is still light out.  Finally lets you have the pit to hit into.  When I arrived I wasn't sure if this was a driving range or an excavation site for a mastodon or a strip mining operation.  There is no grass on the range. Just mud.  Makes it a good tool to figure out how far you actually are hitting the ball.  Oh, they are cash only also.\\\\n\\\\nBottom line, this place sucks.  The best hope is that the owner sells it to someone that actually wants to make money and service golfers in Pittsburgh.\"}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dataset[\"train\"][10]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "6dc45997-e391-456f-b0b9-d3193b0f6a9d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "import pandas as pd\n",
+    "import datasets\n",
+    "from IPython.display import display, HTML"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "9e2ecebb-d5d1-456d-967c-842a79fdd622",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def show_random_elements(dataset, num_examples=10):\n",
+    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
+    "    picks = []\n",
+    "    for _ in range(num_examples):\n",
+    "        pick = random.randint(0, len(dataset)-1)\n",
+    "        while pick in picks:\n",
+    "            pick = random.randint(0, len(dataset)-1)\n",
+    "        picks.append(pick)\n",
+    "    \n",
+    "    df = pd.DataFrame(dataset[picks])\n",
+    "    for column, typ in dataset.features.items():\n",
+    "        if isinstance(typ, datasets.ClassLabel):\n",
+    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
+    "    display(HTML(df.to_html()))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "1af560b6-7d21-499e-9b82-114be371a98a",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>label</th>\n",
+       "      <th>text</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>4 stars</td>\n",
+       "      <td>Great pizza and Sweet Chili wings in The Brewer's Cafe. My wife and I ordered their pizza. I have tried MANY pizzas. My wife and I were pleasantly surprised! My only complaints about this place would be that the menu is just a tad too small. Another tip: Anything fried comes out PIPING HOT!!! ;-)</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2 star</td>\n",
+       "      <td>This is the exact reason why Yelp at times is not a credible resource. You see legitimate users with legitimate reviews frequently get filtered, and you see 4-5 star ratings for restaurants that provide a low quality product. \\n\\nFor years, I kept reading on Yelp that \\\"Blueberry Hill\\\" has the \\\"greatest hand breaded chicken fried steak in all of Las Vegas\\\". There was nothing true about that statement.  The chicken fried steak was nothing more than the processed precooked version that you get at Costco and Smart N Final. All they do is reheat it and serve. I was very disappointed on account of all the good reviews on yelp about how the steak was a \\\"real sirloin that was hand battered\\\". \\n\\nThe side salad was merely lettuce with 1 ring of red onion, served in a beaten up plastic bowl. No tomatoes, no cheese, no croutons...nothing. The Ranch dressing tasted like water mixed with mayonnaise.  \\n\\nThe biscuit that came with our steaks tasted like garlic bread. \\n\\nThe corn bread that came with our steak never came out. The server forgot to bring it out. We never got it. \\n\\nOn a positive note. The Buser was very hard working and made up for the poor quality service that the server provided. The server was not rude, but she was very forgetful and never came around one time to check on us. You need to flag down the server to get her attention. The Buser compensated for the lack of service by coming around to refill our coffee and water. The server was usually found sitting at the counter and reading a book.\\n\\nWhen I asked the server for Tabasco, she forgot. 10 minutes later I asked her again and she finally got it. When I asked the server for Jam, she forgot. When I asked her for saltine crackers, she forgot. She did bring out the check immediately though before we even touched our food. \\n\\nIn conclusion. The bill came out quicker than the food. The buser was the hardest working guy here. The food was processed and low quality. The prices were pretty high. Yelp has failed us all.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>2 star</td>\n",
+       "      <td>For the $18 ticket price, this aquarium was way smaller than I expected. I wish we had taken our time checking things out. We sped through each exhibit thinking there would be more impressive things ahead, but we reached the end quickly and thought, \\\"Is that it?\\\" My nieces had a blast, but they are 4 and 3 and were also amazed by the monorail.\\n\\nThe jellyfish tank and the stingray pool were cool, but nothing stood out to make this place a must-do. One of the employees was more interested in talking about the Lakers instead of the piranhas.\\n\\nIf you're in the area and you need to entertain some kids, it might be ok. If not, skip it and hit the pool instead.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>4 stars</td>\n",
+       "      <td>This was a last minute, drive by decision that fit our timing on a late weekday afternoon.  What a wonderful surprise!  Our timing must have been good, cuz we got right in with no wait.\\n\\nWe got in on the All You Can Eat sushi lunch for 20 bucks.  The rolls were really good and the staff was super attentive.  \\n\\nTiger roll, Mount Charelston roll - thumbs up.  The spicy scallop?  Seems to be a different standard out west, and not my favorite Tsu Kasa style.  The tuna was sublime.  Seaweed salad - the perfect serving size.  Shrimp Tempura - amazingly perfect.  Tried the Ikura (Salmon Roe) but realized I hadn't quite evolved that far.\\n\\nI would go back to this place again and again.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>5 stars</td>\n",
+       "      <td>Not sure why people are writing bad reviews over things out of the clubs control. Going over a few things \\n\\n1.) the most impressive part of this club is the DJs. I've been here twice and both times every dj playing was spot on. \\n\\n2) I personally like the dress code. An upscale gay bar on the strip is much needed. \\n\\n3) I do wish it wasn't in ballys, it's a bit of an annoyance being right in the middle of the chaos of the strip but overall was easy to get to. \\n\\n4) best advice is to try and get on the guest list. Drinks aren't cheap but this is vegas. Find me a cocktail in a nightclub on the strip under 10 bucks and we can say this place is expensive \\n\\n\\nKeep up the great work!</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>5 stars</td>\n",
+       "      <td>Love the juice bar! Try the kaleaid. They now have bottled cold pressed juice. It's expensive, but so good.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>3 stars</td>\n",
+       "      <td>As someone who doesn't drink I am left to base my review on the ambiance and food.  Despite being in a South Scottsdale strip mall the crowd is fun and engaging.  As has been mentioned by the majority of the other reviews, the food isn't really the draw of this place. Still a decent way to pass an evening with friends watching games.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>4 stars</td>\n",
+       "      <td>One of my favorite places for Indian food. Their lunch buffet selection is good and all of their food is delicious! Staff is very friendly as well.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>4 stars</td>\n",
+       "      <td>I loved my sandwich. The Bobbie tastes like Thanksgiving! It's turkey, w/ stuffing, and cranberry on fresh bread. For reals! Extra yum! ...The two dudes that made our sandwiches were cool.</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>1 star</td>\n",
+       "      <td>Do Not Go here They Just Butcherd my Cuticles the guy didn't speak any English and the owners wife was yelling at me because I was upset with my service.. Then she locked me in the place so I called the police on her and her husband.. They Tryed to Chase Me And Then Locked Me inside.. And I even payed her in full and she didn't even finish her job Because she was yelling so loud in the place.. DO NOT WASTE YOUR TIME AND MONEY..LESSON LEARNED NEVER TO GO TO PLACES LIKE THIS EVER AGAIN..</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "show_random_elements(dataset[\"train\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c9df7cd0-23cd-458f-b2b5-f025c3b9fe62",
+   "metadata": {},
+   "source": [
+    "## 预处理数据\n",
+    "\n",
+    "下载数据集到本地后，使用 Tokenizer 来处理文本，对于长度不等的输入数据，可以使用填充（padding）和截断（truncation）策略来处理。\n",
+    "\n",
+    "Datasets 的 `map` 方法，支持一次性在整个数据集上应用预处理函数。\n",
+    "\n",
+    "下面使用填充到最大长度的策略，处理整个数据集："
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "8bf2b342-e1dd-4ab6-ad57-28eb2513ae38",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
+    "\n",
+    "\n",
+    "def tokenize_function(examples):\n",
+    "    return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n",
+    "\n",
+    "\n",
+    "tokenized_datasets = dataset.map(tokenize_function, batched=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "47a415a8-cd15-4a8c-851b-9b4740ef8271",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>label</th>\n",
+       "      <th>text</th>\n",
+       "      <th>input_ids</th>\n",
+       "      <th>token_type_ids</th>\n",
+       "      <th>attention_mask</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>3 stars</td>\n",
+       "      <td>Visited Hillstone to give them another try a couple weeks ago.  Finally had the time to sit down and write an update.\\n\\nAs usual the food was great.  The service, while better this occasion was still a bit off.  Two examples:\\n\\n1) Length of time for wait staff to revisit our table for refreshments was fairly lengthy in comparison to the old Houston's.\\n\\n2) When asked about the sauce used in the Thai Tuna Roll, the waitress looked very blank and then said it was in the roll. She replied, tuna and some sort of Thai sauce.  Noooo kidding.\\n\\nI will give this place another chance and another star because I really like their food.  I will more than likely return.  However, I will update with new reviews until it's back to the place I remember.</td>\n",
+       "      <td>[101, 159, 26868, 1906, 5377, 4793, 1106, 1660, 1172, 1330, 2222, 170, 2337, 2277, 2403, 119, 4428, 1125, 1103, 1159, 1106, 3465, 1205, 1105, 3593, 1126, 11984, 119, 165, 183, 165, 183, 23390, 4400, 1103, 2094, 1108, 1632, 119, 1109, 1555, 117, 1229, 1618, 1142, 6116, 1108, 1253, 170, 2113, 1228, 119, 1960, 5136, 131, 165, 183, 165, 183, 1475, 114, 16758, 1104, 1159, 1111, 3074, 2546, 1106, 1231, 9356, 2875, 1412, 1952, 1111, 1231, 2087, 21298, 4385, 1108, 6751, 12628, 1107, 7577, 1106, 1103, 1385, 4666, 112, 188, 119, 165, 183, 165, 183, 1477, 114, 1332, 1455, 1164, 1103, ...]</td>\n",
+       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]</td>\n",
+       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...]</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "show_random_elements(tokenized_datasets[\"train\"], num_examples=1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c33d153-f729-4f04-972c-a764c1cbbb8b",
+   "metadata": {},
+   "source": [
+    "### 数据抽样\n",
+    "\n",
+    "使用 1000 个数据样本，在 BERT 上演示小规模训练（基于 Pytorch Trainer）\n",
+    "\n",
+    "`shuffle()`函数会随机重新排列列的值。如果您希望对用于洗牌数据集的算法有更多控制，可以在此函数中指定generator参数来使用不同的numpy.random.Generator。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "a17317d8-3c6a-467f-843d-87491f600db1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
+    "# small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d3b65d63-2d3a-4a56-bc31-6e88a29e9dec",
+   "metadata": {},
+   "source": [
+    "## 微调训练配置\n",
+    "\n",
+    "### 加载 BERT 模型\n",
+    "\n",
+    "警告通知我们正在丢弃一些权重（`vocab_transform` 和 `vocab_layer_norm` 层），并随机初始化其他一些权重（`pre_classifier` 和 `classifier` 层）。在微调模型情况下是绝对正常的，因为我们正在删除用于预训练模型的掩码语言建模任务的头部，并用一个新的头部替换它，对于这个新头部，我们没有预训练的权重，所以库会警告我们在用它进行推理之前应该对这个模型进行微调，而这正是我们要做的事情。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "4d2af4df-abd4-4a4b-94b6-b0e7375304ed",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
+      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from transformers import AutoModelForSequenceClassification\n",
+    "\n",
+    "model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-cased\", num_labels=5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b44014df-b52c-4c72-9e9f-54424725a473",
+   "metadata": {},
+   "source": [
+    "### 训练超参数（TrainingArguments）\n",
+    "\n",
+    "完整配置参数与默认值：https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments\n",
+    "\n",
+    "源代码定义：https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/training_args.py#L161\n",
+    "\n",
+    "**最重要配置：模型权重保存路径(output_dir)**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "98c01d5c-de72-4ff0-b11d-e07ac5346888",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# from transformers import TrainingArguments\n",
+    "\n",
+    "# model_dir = \"models/bert-base-cased\"\n",
+    "\n",
+    "# # logging_steps 默认值为500，根据我们的训练数据和步长，将其设置为100\n",
+    "# training_args = TrainingArguments(output_dir=f\"{model_dir}/test_trainer\",\n",
+    "#                                   logging_dir=f\"{model_dir}/test_trainer/runs\",\n",
+    "#                                   logging_steps=100)\n",
+    "# # 完整的超参数配置\n",
+    "# print(training_args)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0ce03480-3aaa-48ea-a0c6-a177b8d8e34f",
+   "metadata": {
+    "collapsed": true,
+    "jupyter": {
+     "outputs_hidden": true
+    }
+   },
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7ebd3365-d359-4ab4-a300-4717590cc240",
+   "metadata": {},
+   "source": [
+    "### 训练过程中的指标评估（Evaluate)\n",
+    "\n",
+    "**[Hugging Face Evaluate 库](https://huggingface.co/docs/evaluate/index)** 支持使用一行代码，获得数十种不同领域（自然语言处理、计算机视觉、强化学习等）的评估方法。 当前支持 **完整评估指标：https://huggingface.co/evaluate-metric**\n",
+    "\n",
+    "训练器（Trainer）在训练过程中不会自动评估模型性能。因此，我们需要向训练器传递一个函数来计算和报告指标。 \n",
+    "\n",
+    "Evaluate库提供了一个简单的准确率函数，您可以使用`evaluate.load`函数加载"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "2a8ef138-5bf2-41e5-8c68-df8e11f4e98f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import evaluate\n",
+    "\n",
+    "metric = evaluate.load(\"accuracy\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "70d406c0-56d0-4a54-9c6c-e126ab7f5254",
+   "metadata": {},
+   "source": [
+    "\n",
+    "接着，调用 `compute` 函数来计算预测的准确率。\n",
+    "\n",
+    "在将预测传递给 compute 函数之前，我们需要将 logits 转换为预测值（**所有Transformers 模型都返回 logits**）。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "f46d2e59-1ebf-43d2-bc86-6b57a4d24d19",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def compute_metrics(eval_pred):\n",
+    "    logits, labels = eval_pred\n",
+    "    predictions = np.argmax(logits, axis=-1)\n",
+    "    return metric.compute(predictions=predictions, references=labels)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e2feba67-9ca9-4793-9a15-3eaa426df2a1",
+   "metadata": {},
+   "source": [
+    "#### 训练过程指标监控\n",
+    "\n",
+    "通常，为了监控训练过程中的评估指标变化，我们可以在`TrainingArguments`指定`evaluation_strategy`参数，以便在 epoch 结束时报告评估指标。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "id": "afaaee18-4986-4e39-8ad9-b8d413ab4cd1",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from transformers import TrainingArguments, Trainer\n",
+    "model_dir = \"models/bert-base-cased\"\n",
+    "batch_size = 14\n",
+    "\n",
+    "training_args = TrainingArguments(\n",
+    "    output_dir=f\"{model_dir}/test_trainer\",\n",
+    "    evaluation_strategy=\"epoch\", \n",
+    "    logging_dir=f\"{model_dir}/test_trainer/runs\",\n",
+    "    logging_steps=500,\n",
+    "    save_total_limit=3,\n",
+    "    per_device_train_batch_size=batch_size,\n",
+    "    per_device_eval_batch_size=batch_size,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d47d6981-e444-4c0f-a7cb-dd7f2ba8df12",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "source": [
+    "## 开始训练\n",
+    "\n",
+    "### 实例化训练器（Trainer）\n",
+    "\n",
+    "`kernel version` 版本问题：暂不影响本示例代码运行"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 63,
+   "id": "ca1d12ac-89dc-4c30-8282-f859724c0062",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n",
+    "small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=training_args,\n",
+    "    train_dataset=tokenized_datasets[\"train\"],\n",
+    "    eval_dataset=small_eval_dataset,\n",
+    "    compute_metrics=compute_metrics,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 64,
+   "id": "9b3c069d-a0dc-4f43-aea0-6cb8799643f3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# trainer.args"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "449eb845-cff7-40ba-8915-38de79248840",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a833e0db-1168-4a3c-8b75-bfdcef8c5157",
+   "metadata": {},
+   "source": [
+    "## 使用 nvidia-smi 查看 GPU 使用\n",
+    "\n",
+    "为了实时查看GPU使用情况，可以使用 `watch` 指令实现轮询：`watch -n 1 nvidia-smi`:\n",
+    "\n",
+    "```shell\n",
+    "Every 1.0s: nvidia-smi                                                   Wed Dec 20 14:37:41 2023\n",
+    "\n",
+    "Wed Dec 20 14:37:41 2023\n",
+    "+---------------------------------------------------------------------------------------+\n",
+    "| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |\n",
+    "|-----------------------------------------+----------------------+----------------------+\n",
+    "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
+    "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
+    "|                                         |                      |               MIG M. |\n",
+    "|=========================================+======================+======================|\n",
+    "|   0  Tesla T4                       Off | 00000000:00:0D.0 Off |                    0 |\n",
+    "| N/A   64C    P0              69W /  70W |   6665MiB / 15360MiB |     98%      Default |\n",
+    "|                                         |                      |                  N/A |\n",
+    "+-----------------------------------------+----------------------+----------------------+\n",
+    "\n",
+    "+---------------------------------------------------------------------------------------+\n",
+    "| Processes:                                                                            |\n",
+    "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
+    "|        ID   ID                                                             Usage      |\n",
+    "|=======================================================================================|\n",
+    "|    0   N/A  N/A     18395      C   /root/miniconda3/bin/python                6660MiB |\n",
+    "+---------------------------------------------------------------------------------------+\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 65,
+   "id": "accfe921-471d-481a-96da-c491cdebad0c",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <div>\n",
+       "      \n",
+       "      <progress value='46431' max='46431' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
+       "      [46431/46431 17:33:04, Epoch 3/3]\n",
+       "    </div>\n",
+       "    <table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       " <tr style=\"text-align: left;\">\n",
+       "      <th>Epoch</th>\n",
+       "      <th>Training Loss</th>\n",
+       "      <th>Validation Loss</th>\n",
+       "      <th>Accuracy</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>1</td>\n",
+       "      <td>0.727000</td>\n",
+       "      <td>0.694410</td>\n",
+       "      <td>0.703000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>2</td>\n",
+       "      <td>0.633200</td>\n",
+       "      <td>0.691635</td>\n",
+       "      <td>0.710000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <td>3</td>\n",
+       "      <td>0.528600</td>\n",
+       "      <td>0.732436</td>\n",
+       "      <td>0.711000</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table><p>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "IOPub message rate exceeded.\n",
+      "The Jupyter server will temporarily stop sending output\n",
+      "to the client in order to avoid crashing it.\n",
+      "To change this limit, set the config variable\n",
+      "`--ServerApp.iopub_msg_rate_limit`.\n",
+      "\n",
+      "Current values:\n",
+      "ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
+      "ServerApp.rate_limit_window=3.0 (secs)\n",
+      "\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n",
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "TrainOutput(global_step=46431, training_loss=0.6399251460484753, metrics={'train_runtime': 63185.9462, 'train_samples_per_second': 30.861, 'train_steps_per_second': 0.735, 'total_flos': 5.130803778048e+17, 'train_loss': 0.6399251460484753, 'epoch': 3.0})"
+      ]
+     },
+     "execution_count": 65,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "trainer.train(False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 66,
+   "id": "6d581099-37a4-4470-b051-1ada38554089",
+   "metadata": {
+    "editable": true,
+    "slideshow": {
+     "slide_type": ""
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "small_test_dataset = tokenized_datasets[\"test\"].shuffle(seed=64).select(range(100))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 67,
+   "id": "ffb47eab-1370-491e-8a84-6d5347a350b2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/lib/python3.9/dist-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.\n",
+      "  warnings.warn('Was asked to gather along dimension 0, but all '\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <div>\n",
+       "      \n",
+       "      <progress value='3' max='3' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
+       "      [3/3 00:00]\n",
+       "    </div>\n",
+       "    "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'eval_loss': 0.8645618557929993,\n",
+       " 'eval_accuracy': 0.65,\n",
+       " 'eval_runtime': 1.3182,\n",
+       " 'eval_samples_per_second': 75.861,\n",
+       " 'eval_steps_per_second': 2.276,\n",
+       " 'epoch': 3.0}"
+      ]
+     },
+     "execution_count": 67,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "trainer.evaluate(small_test_dataset)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "27a55686-7c43-4ab8-a5cd-0e77f14c7c52",
+   "metadata": {},
+   "source": [
+    "### 保存模型和训练状态\n",
+    "\n",
+    "- 使用 `trainer.save_model` 方法保存模型，后续可以通过 from_pretrained() 方法重新加载\n",
+    "- 使用 `trainer.save_state` 方法保存训练状态"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 68,
+   "id": "ad0cbc14-9ef7-450f-a1a3-4f92b6486f41",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.save_model(f\"{model_dir}/finetuned-trainer\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 69,
+   "id": "badf5868-2847-439d-a73e-42d1cca67b5e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer.save_state()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "61828934-01da-4fc3-9e75-8d754c25dfbc",
+   "metadata": {},
+   "source": [
+    "## Homework: 使用完整的 YelpReviewFull 数据集训练，对比看 Acc 最高能到多少"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 74,
+   "id": "6ee2580a-7a5a-46ae-a28b-b41e9e838eb1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "model.safetensors: 100%|██████████| 433M/433M [00:15<00:00, 28.0MB/s] \n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "CommitInfo(commit_url='https://huggingface.co/yqzhangjx/bert-base-cased-for-yelp/commit/ef8247a2eb2c3e93a70f0198591833256f6d197c', commit_message='Upload BertForSequenceClassification', commit_description='', oid='ef8247a2eb2c3e93a70f0198591833256f6d197c', pr_url=None, pr_revision=None, pr_num=None)"
+      ]
+     },
+     "execution_count": 74,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model.push_to_hub(\"yqzhangjx/bert-base-cased-for-yelp\", token=\"XXX\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "478f8d8e-2597-4a6c-a84c-d66e3d231e1d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "561af3da-d720-4478-99de-b72d7419fb37",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}