added new jupyter notebook

00-HF-Setup.ipynb

@@ -0,0 +1,303 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "131de9a7-0245-4e12-9554-c8f91eec2d21",
+   "metadata": {},
+   "source": [
+    "# Hugging Face Setup\n",
+    "\n",
+    "Let's quickly make sure HF is set up and that you are able to access downloads from Hugging Face Hub using your Token."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e0b59f0-1180-4297-94dc-b26115761264",
+   "metadata": {},
+   "source": [
+    "## Python Libraries Install:\n",
+    "\n",
+    "Note that we use various versions of these libraries throughout the course, make sure to watch the video to know which version to use!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9a2f2dca-fefc-4f36-91f3-4dbcc07345a6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !pip install transformers diffusers datasets evaluate accelerate"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "ac9d74b6-a136-4858-ab8b-1e7791993380",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import notebook_login"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "0899dcf9-7504-4097-beb8-c42a519305c3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "92c0780a54384af9ae5d07a3f5d4766d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "notebook_login()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "7d706a5b-456f-41d1-83a9-98fa397ce90a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import scan_cache_dir\n",
+    "\n",
+    "hf_cache_info = scan_cache_dir()\n",
+    "# print(hf_cache_info)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0d0cc79a-6eb3-44c5-b98b-5deb40acd62b",
+   "metadata": {},
+   "source": [
+    "When you work with Hugging Face's Python libraries, such as the `transformers` library, you'll often download pre-trained models and datasets. These downloaded files are stored locally on your machine to avoid repeated downloads and to ensure quick access in future uses. Let's explore where and how these files are stored.\n",
+    "\n",
+    "## Where Are Hugging Face Models Stored?\n",
+    "\n",
+    "By default, Hugging Face stores downloaded models in a directory under your home directory. Specifically, it uses a hidden folder named `.cache`. The typical path looks like this:\n",
+    "\n",
+    "- On Unix-based systems (Linux, macOS):\n",
+    "  ```\n",
+    "  ~/.cache/huggingface/ \n",
+    "  ```\n",
+    "\n",
+    "- On Windows systems:\n",
+    "  ```\n",
+    "  C:\\Users\\<YourUsername>\\.cache\\huggingface\\ \n",
+    "  ```\n",
+    " \n",
+    "**NOTE - .cache is hidden by default! You will need to set hidden files viewable to see it!\n",
+    "\n",
+    "----\n",
+    "\n",
+    "Hidden directories are often used to store configuration files and caches. These directories are typically not shown in default file explorer views. Here’s how you can view hidden directories on different operating systems:\n",
+    "\n",
+    "## Viewing Hidden Directories on Different Operating Systems\n",
+    "\n",
+    "### macOS\n",
+    "\n",
+    "On macOS, hidden directories and files (those starting with a dot, such as `.cache`) can be made visible in Finder:\n",
+    "\n",
+    "1. **Using Finder:**\n",
+    "   - Open Finder.\n",
+    "   - Press `Command + Shift + .` (period). This will toggle the visibility of hidden files and directories.\n",
+    "\n",
+    "2. **Using Terminal:**\n",
+    "   - Open Terminal.\n",
+    "   - To list hidden files in a directory, use the following command:\n",
+    "     ```bash\n",
+    "     ls -la\n",
+    "     ```\n",
+    "   - The `-a` flag shows all files, including hidden ones, and the `-l` flag gives a detailed listing.\n",
+    "\n",
+    "### Linux\n",
+    "\n",
+    "On Linux, hidden files and directories can be viewed in the file manager or terminal:\n",
+    "\n",
+    "1. **Using File Manager (e.g., Nautilus):**\n",
+    "   - Open your file manager.\n",
+    "   - Press `Ctrl + H`. This will toggle the visibility of hidden files and directories.\n",
+    "\n",
+    "2. **Using Terminal:**\n",
+    "   - Open Terminal.\n",
+    "   - To list hidden files in a directory, use the following command:\n",
+    "     ```bash\n",
+    "     ls -la\n",
+    "     ```\n",
+    "   - The `-a` flag shows all files, including hidden ones, and the `-l` flag gives a detailed listing.\n",
+    "\n",
+    "### Windows\n",
+    "\n",
+    "On Windows, hidden files and directories can be viewed in File Explorer:\n",
+    "\n",
+    "1. **Using File Explorer:**\n",
+    "   - Open File Explorer.\n",
+    "   - Click on the `View` tab at the top.\n",
+    "   - Check the box for `Hidden items` in the Show/hide group. This will toggle the visibility of hidden files and directories.\n",
+    "\n",
+    "2. **Using Command Prompt:**\n",
+    "   - Open Command Prompt.\n",
+    "   - To list hidden files in a directory, use the following command:\n",
+    "     ```cmd\n",
+    "     dir /a\n",
+    "     ```\n",
+    "   - The `/a` flag lists all files, including hidden ones.\n",
+    "\n",
+    "## Summary\n",
+    "\n",
+    "Viewing hidden directories on different operating systems is straightforward:\n",
+    "\n",
+    "- **macOS:** Press `Command + Shift + .` in Finder or use `ls -la` in Terminal.\n",
+    "- **Linux:** Press `Ctrl + H` in the file manager or use `ls -la` in Terminal.\n",
+    "- **Windows:** Check `Hidden items` in File Explorer’s View tab or use `dir /a` in Command Prompt.\n",
+    "\n",
+    "These methods allow you to easily access and manage hidden files and directories on your system.\n",
+    "\n",
+    "----\n",
+    "\n",
+    "**Ok, let's move on, back to Hugging Face topics!**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b5346ae4-c22c-4bc6-ab7d-21124699bce8",
+   "metadata": {},
+   "source": [
+    "The Hugging Face Python libraries store downloaded models in a centralized cache directory. This cache system is designed to be shared across various libraries that depend on the Hugging Face Hub. Here is a detailed explanation of where and how these models are stored:\n",
+    "\n",
+    "## Cache Directory Structure\n",
+    "\n",
+    "The cache directory is typically located in the user's home directory, but it can be customized using the `cache_dir` argument in methods or by setting the `HF_HOME` or `HF_HUB_CACHE` environment variables. The structure of the cache directory is as follows:\n",
+    "\n",
+    "```\n",
+    "<CACHE_DIR>\n",
+    "├─ <MODELS>\n",
+    "├─ <DATASETS>\n",
+    "├─ <SPACES>\n",
+    "```\n",
+    "\n",
+    "Within these main folders, the cache is further organized by repository type, namespace (if applicable), and repository name. For example:\n",
+    "\n",
+    "```\n",
+    "<CACHE_DIR>\n",
+    "├─ models--julien-c--EsperBERTo-small\n",
+    "├─ models--lysandrejik--arxiv-nlp\n",
+    "├─ models--bert-base-cased\n",
+    "├─ datasets--glue\n",
+    "├─ datasets--huggingface--DataMeasurementsFiles\n",
+    "├─ spaces--dalle-mini--dalle-mini\n",
+    "```\n",
+    "\n",
+    "## Detailed Folder Structure\n",
+    "\n",
+    "Each repository folder contains subfolders that store different types of files, such as references, blobs, and snapshots. Here is an example of the folder structure for a dataset:\n",
+    "\n",
+    "```\n",
+    "<CACHE_DIR>\n",
+    "├─ datasets--glue\n",
+    "│   ├─ refs\n",
+    "│   ├─ blobs\n",
+    "│   ├─ snapshots\n",
+    "```\n",
+    "\n",
+    "## Managing the Cache\n",
+    "\n",
+    "### Scanning the Cache\n",
+    "\n",
+    "To manage and inspect the cache, you can use the `huggingface-cli` tool or the `scan_cache_dir` function from the `huggingface_hub` library. This allows you to see which repositories and revisions are taking up disk space. For example:\n",
+    "\n",
+    "```python\n",
+    "from huggingface_hub import scan_cache_dir\n",
+    "\n",
+    "hf_cache_info = scan_cache_dir()\n",
+    "print(hf_cache_info)\n",
+    "```\n",
+    "\n",
+    "This will return an `HFCacheInfo` object containing details about the cached repositories, their sizes, and any warnings about corrupted caches.\n",
+    "\n",
+    "### Example Command\n",
+    "\n",
+    "Using the `huggingface-cli` to scan the cache:\n",
+    "\n",
+    "```bash\n",
+    "huggingface-cli scan-cache\n",
+    "```\n",
+    "\n",
+    "This command will output a detailed report of the cache, including repository IDs, types, sizes, and paths.\n",
+    "\n",
+    "## Customizing the Cache Directory\n",
+    "\n",
+    "You can customize the cache directory by setting the `cache_dir` argument in methods or by using environment variables. For example:\n",
+    "\n",
+    "```python\n",
+    "from huggingface_hub import cached_assets_path\n",
+    "\n",
+    "path = cached_assets_path(library_name=\"datasets\", namespace=\"SQuAD\", subfolder=\"download\")\n",
+    "print(path)\n",
+    "```\n",
+    "\n",
+    "This will return the path to the cached assets for the specified library, namespace, and subfolder.\n",
+    "\n",
+    "## Conclusion\n",
+    "\n",
+    "The Hugging Face cache system is designed to efficiently store and manage models, datasets, and other resources. By understanding the structure and management tools available, users can effectively control their cache usage and ensure optimal performance.\n",
+    "\n",
+    "For more detailed information, you can refer to the Hugging Face documentation on managing the cache system[1][3][5][7].\n",
+    "\n",
+    "Citations:\n",
+    "[1] https://huggingface.co/docs/huggingface_hub/guides/manage-cache\n",
+    "[2] https://discuss.huggingface.co/t/model-caching-and-locking/44152\n",
+    "[3] https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache\n",
+    "[4] https://huggingface.co/docs/hub/en/models\n",
+    "[5] https://huggingface.co/docs/huggingface_hub/package_reference/cache\n",
+    "[6] https://huggingface.co/docs/hub/en/models-libraries\n",
+    "[7] https://huggingface.co/docs/huggingface_hub/en/package_reference/cache\n",
+    "[8] https://huggingface.co/docs/hub/en/models-adding-libraries\n",
+    "[9] https://discuss.huggingface.co/t/how-to-save-my-model-to-use-it-later/20568\n",
+    "[10] https://huggingface.co/docs/transformers/en/main_classes/model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d40ac023-8adb-4206-8954-457f352b4d76",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}