File size: 2,420 Bytes
8933f91 db4aeca e5d7840 db4aeca 8933f91 6058b6f 8933f91 6058b6f 8933f91 76d3ff4 8933f91 6058b6f 8933f91 6058b6f 8933f91 6058b6f 8933f91 76d3ff4 8933f91 6058b6f 76d3ff4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
# SAFE Example Submission with Custom Environment
This is an example of a submission that includes a custom environment.
So instead of using the default environment of the submission, we first create a custom environment, package it with conda-pack and then push to the model hub in addition to the rest of the files.
Here are the steps:
1. ## Determine the environment that you want and save to the `requirements-custom.txt` file.
The best way is to start from the base competition image, create a new environment, test it with your model and use `pip freeze` to export it.
To start the container:
```bash
docker run -it --rm --gpus all -v /path/to/local:/app/custom-env huggingface/competitions:latest bash
```
to create an environment:
```bash
conda init
source ~/.bashrc
conda create -n custom python=3.10 -y
conda activate custom
```
Then you can install packages with pip, conda, etc.
2. ## Export the environment with `conda-pack` to a `.tar.gz` file.
Run `docker-compose up`. This will pull the base competition image, copy the files into the container and create a new environment save it.
Change the local place where the volume `/app/custom-env` mounts in the docker-compose file. It will run `make_custom_env.sh` inside the container.
```bash
conda init
source ~/.bashrc
conda create -n custom python=3.10 -y
conda activate custom
pip install -r requirements-custom.txt
export OUTPUT=/app/custom-env/custom.tar.gz
conda deactivate
pip install conda-pack
conda-pack -n custom -o $OUTPUT
```
3. ## Upload the enviroment. Since the environment file is large.
It's better to upload it seperately using fast HF_TRANSFER. Make sure to install `pip install huggingface_hub[hf_transfer]`.
```bash
bash upload_env_file
```
Make sure to change the path to your env file in the script. This is what the script does:
```bash
ENV_FILE=/disk1/kirill/custom-env/custom.tar.gz
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli upload safe-challenge/safe-example-submission-custom-env $ENV_FILE custom.tar.gz
```
5. ## During runtime, `script.py` unpacks the environment to run your model in your custom environment
Rename your actual script to `_script.py`. `script.py` just calls `bash run.sh` to do all the steps:
```bash
echo "uncompressing environment"
mkdir -p custom
tar -xzf custom.tar.gz -C custom
source custom/bin/activate
echo "unpacking"
conda-unpack
echo "running"
python _script.py
```
|