File size: 5,825 Bytes
b585c7f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 |
## h2oGPT Installation Help
The following sections describe how to get a working Python environment on a Linux system.
### Install for A100+
E.g. for Ubuntu 20.04, install driver if you haven't already done so:
```bash
sudo apt-get update
sudo apt-get -y install nvidia-headless-535-server nvidia-fabricmanager-535 nvidia-utils-535-server
# sudo apt-get -y install nvidia-headless-no-dkms-535-servers
```
Note that if you run the preceding commands, you don't need to use the NVIDIA developer downloads in the following sections.
### Install CUDA Toolkit
If happy with above drivers, then just get run local file for [CUDA 11.8](https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local):
```bash
wget wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run
```
only choose to install toolkit and do not replace existing `/usr/local/cuda` link if you already have one.
If instead, you want full deb CUDA [install cuda coolkit](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local). Pick deb local, e.g. for Ubuntu:
```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda
```
Then set the system up to use the freshly installed CUDA location:
```bash
echo "export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/local/cuda/lib64/" >> ~/.bashrc
echo "export CUDA_HOME=/usr/local/cuda" >> ~/.bashrc
echo "export PATH=\$PATH:/usr/local/cuda/bin/" >> ~/.bashrc
source ~/.bashrc
```
Then reboot the machine, to get everything sync'ed up on restart.
```bash
sudo reboot
```
### Compile bitsandbytes
For fast 4-bit and 8-bit training, you need to use [bitsandbytes](https://github.com/TimDettmers/bitsandbytes/tree/main#readme). Note that [compiling bitsandbytes](https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md) is only required if you have a different CUDA version from the ones built into the [bitsandbytes PyPI package](https://pypi.org/project/bitsandbytes/),
which includes CUDA 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8, 12.0, and 12.1. In the following example, bitsandbytes is compiled for CUDA 12.1:
```bash
git clone http://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
git checkout 7c651012fce87881bb4e194a26af25790cadea4f
CUDA_VERSION=121 make cuda12x
CUDA_VERSION=121 python setup.py install
cd ..
```
### Install NVIDIA GPU Manager on systems with multiple A100 or H100 GPUs
To install NVIDIA GPU Manager, run the following:
```bash
sudo apt-key del 7fa2af80
distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get install -y datacenter-gpu-manager
# if use 535 drivers, then use 535 below
sudo apt-get install -y libnvidia-nscq-535
sudo systemctl --now enable nvidia-dcgm
dcgmi discovery -l
```
For more information, see the official [GPU Manager user guide](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html).
### Install and run NVIDIA Fabric Manager on systems with multiple A100 or H100 GPUs
To install the CUDA drivers for NVIDIA Fabric Manager, run the following:
```bash
sudo apt-get install -y cuda-drivers-fabricmanager
```
Once you've installed Fabric Manager and rebooted your system, run the following to start the NVIDIA Fabric Manager service:
```bash
sudo systemctl --now enable nvidia-dcgm
dcgmi discovery -l
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager
```
For more information, see the official [Fabric Manager user guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html).
### Optional: Use TensorBoard to inspect training
You can use [TensorBoard](https://www.tensorflow.org/tensorboard/get_started) to inspect the training process. To launch TensorBoard and instruct it to read event files from the `runs/` directory, use the following command:
```bash
tensorboard --logdir=runs/
```
For more information, see [TensorBoard usage](https://github.com/tensorflow/tensorboard/blob/master/README.md#usage).
### Flash Attention
**Update:** Flash attention specifics are no longer needed. For more information, see https://github.com/h2oai/h2ogpt/issues/128.
To use flash attention with LLaMa, need cuda 11.7 so flash attention module compiles against torch.
E.g. for Ubuntu, one goes to [cuda toolkit](https://developer.nvidia.com/cuda-11-7-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local), then:
```bash
wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
sudo bash ./cuda_11.7.0_515.43.04_linux.run
```
Then No for symlink change, say continue (not abort), accept license, keep only toolkit selected, select install.
If cuda 11.7 is not your base installation, then when doing pip install -r requirements.txt do instead:
```bash
CUDA_HOME=/usr/local/cuda-11.8 pip install -r reqs_optional/requirements_optional_flashattention.txt
```
|