|
--- |
|
title: Ortha |
|
emoji: 🖼 |
|
colorFrom: purple |
|
colorTo: red |
|
sdk: gradio |
|
sdk_version: 4.26.0 |
|
app_file: app.py |
|
pinned: false |
|
license: apache-2.0 |
|
--- |
|
|
|
# Orthogonal Adaptation |
|
|
|
## 🔧 Dependencies and Installation |
|
|
|
- Python >= 3.9 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html)) |
|
- Diffusers==0.19.3 |
|
- XFormer (is recommend to save memory) |
|
|
|
## ⏬ Pretrained Model and Data Preparation |
|
|
|
### Pretrained Model Preparation |
|
|
|
We adopt the [ChilloutMix](https://civitai.com/models/6424/chilloutmix) fine-tuned model for generating human subjects. |
|
|
|
```bash |
|
git clone https://github.com/TencentARC/Mix-of-Show.git |
|
|
|
cd experiments/pretrained_models |
|
|
|
# Diffusers-version ChilloutMix |
|
git-lfs clone https://huggingface.co/windwhinny/chilloutmix.git |
|
``` |
|
<!-- ### Data Preparation |
|
|
|
Note: Data selection and tagging are important in single-concept tuning. We strongly recommend checking the data processing in [sd-scripts](https://github.com/kohya-ss/sd-scripts). **In our ED-LoRA, we do not require any regularization dataset.** The detailed dataset preparation steps can refer to [Dataset.md](docs/Dataset.md). Our preprocessed data used in this repo is available at [Google Drive](https://drive.google.com/file/d/1O5oev8861N_KmKtqefb45l3SiSblbo5O/view?usp=sharing). --> |
|
|
|
## 🕹️ Single-Client Concept Tuning |
|
### Step 0: Data selection and Tagging for a single concept |
|
Data selection and tagging are crucial in single-concept tuning. We strongly recommend checking the data processing in [sd-scripts](https://github.com/kohya-ss/sd-scripts). **In our ED-LoRA, we do not require any regularization dataset.** |
|
|
|
1. **Collect Images**: Gather 5-10 images of the concept you want to customize and place them inside a single folder located at `/single-concept/data/yourconceptname/image`. Ensure the images are consistent but also varied in appearance to prevent overfitting. |
|
|
|
2. **Create Captions**: Write captions for each image you collected. Save these captions as text files in the `/single-concept/data/yourconceptname/caption` directory. |
|
|
|
3. **Generate Masks**: To further improve the understanding of the concept, save masks of each image in the `/single-concept/data/yourconceptname/mask` directory. Use the `data_processing.ipynb` notebook for this step. |
|
|
|
4. **Create Data Configs**: In the `/single-concept/data_configs` directory, create a JSON file that summarizes the files you just created. The file name could be `yourconceptname.json`. |
|
|
|
|
|
### Step 1: Modify the Config at `/single-concept/train_configs` |
|
|
|
Before tuning, it is essential to specify the data paths and adjust certain hyperparameters in the corresponding config file. Below are some basic config settings to be modified: |
|
|
|
- **Concept List**: The data config that you just created should be referenced from 'concept_list'. |
|
- **Validation Prompt**: You might want to set a proper validation prompt in `single-concept/validation_prompts` to visualize the single-concept sampling. |
|
|
|
|
|
```yaml |
|
manual_seed: 1234 # this seed determines choice of columns from orthogonal basis (set differently for each concept) |
|
|
|
datasets: |
|
train: |
|
# Concept data config |
|
concept_list: single-concept/data_configs/hina_amano.json |
|
replace_mapping: |
|
<TOK>: <hina1> <hina2> # concept new token |
|
val_vis: |
|
# Validation prompt for visualization during tuning |
|
prompts: single-concept/validation_prompts/characters/test_girl.txt |
|
replace_mapping: |
|
<TOK>: <hina1> <hina2> # Concept new token |
|
|
|
models: |
|
enable_edlora: true # true means ED-LoRA, false means vallina LoRA |
|
new_concept_token: <hina1>+<hina2> # Concept new token, use "+" to connect |
|
initializer_token: <rand-0.013>+girl |
|
# Init token, only need to revise the later one based on the semantic category of given concept |
|
|
|
val: |
|
val_during_save: true # When saving checkpoint, visualize sample results. |
|
compose_visualize: true # Compose all samples into a large grid figure for visualization |
|
``` |
|
|
|
### Step 2: Start Tuning |
|
|
|
We tune each concept with 2 A100 GPU. Similar to LoRA, community user can enable gradient accumulation, xformer, gradient checkpoint for tuning on one GPU. |
|
|
|
```bash |
|
accelerate launch train_edlora.py -opt single-concept/0005_lebron_ortho.yml |
|
``` |
|
The LoRA weights for the single concept are saved inside the `/experiments/single-concept` folder under your concept name folder |
|
|
|
### Step 3: Single-concept Sampling |
|
To sample an image from your trained weights from the last step, specify the model path in the sample config (located in `/single-concept/sample_configs`) and run the following command: |
|
|
|
```bash |
|
python test_edlora.py -opt single-concept/sample_configs/8101_EDLoRA_potter_Cmix_B4_Repeat500.yml |
|
``` |
|
|
|
## 🕹️ Merging LoRAs |
|
|
|
### Step 1: Collect Concept Models |
|
|
|
Collect all concept models you want to extend the pretrained model and modify the config in `/multi-concept/merge_configs` accordingly. |
|
|
|
```yaml |
|
[ |
|
{ |
|
"lora_path": "experiments/0022_elsa_ortho/models/edlora_model-latest.pth", |
|
"unet_alpha": 1.8, |
|
"text_encoder_alpha": 1.8, |
|
"concept_name": "<elsa1> <elsa2>" |
|
}, |
|
{ |
|
"lora_path": "experiments/0023_moana_ortho/models/edlora_model-latest.pth", |
|
"unet_alpha": 1.8, |
|
"text_encoder_alpha": 1.8, |
|
"concept_name": "<moana1> <moana2>" |
|
} |
|
... # keep adding new concepts for extending the pretrained models |
|
] |
|
``` |
|
|
|
### Step 2: Weight Fusion |
|
Specify which merge config you are using inside the `fuse.sh` file, and then run: |
|
|
|
```bash |
|
bash fuse.sh |
|
``` |
|
The merged weights are now saved in the `/experiments/multi-concept` directory. This process is almost instant. |
|
|
|
### Step 3: Sample |
|
|
|
**Regionally controllable multi-concept sampling:** |
|
|
|
We utilize regionally controllable sampling from Mix-of-Show to enable multi-concept generation. Adding openpose conditioning greatly increases the reliability of generations. |
|
|
|
Define which fused model inside `/experiments/multi-concept` you are going to use, and specify the keypose condition in `/multi-concept/pose_data` if needed. Also, modify the context prompts and regional prompts. Then run: |
|
|
|
```bash |
|
bash regionally_sample.sh |
|
``` |
|
The samples from the multi-concept generation will now be stored in the `/results` folder. |
|
|
