tuners/clay_style_edit/README.md

---
frameworks:
- Pytorch
license: apache-2.0
tasks:
- efficient-diffusion-tuning
---

<p align="center">

  <h2 align="center">clay_style_edit</h2>
  <p align="center">
    <br>
        <a href="https://github.com/modelscope/scepter/"><img src="https://img.shields.io/badge/powered by-scepter-6FEBB9.svg"></a>
    <br>
  </p>

## Model Introduction
Transfer images into clay style

## Model Parameters
<table>
<thead>
  <tr>
    <th rowspan="2">Base Model</th>
    <th rowspan="2">Tuner Type</th>
    <th colspan="4">Training Parameters</th>
  </tr>
  <tr>
    <th>Batch Size</th>
    <th>Epochs</th>
    <th>Learning Rate</th>
    <th>Resolution</th>
  </tr>
</thead>
<tbody align="center">
  <tr>
    <td rowspan="8">EDIT</td>
    <td>LORA</td>
    <td>1</td>
    <td>50</td>
    <td>0.0001</td>
    <td>[512, 512]</td>
  </tr>
</tbody>
</table>


<table>
<thead>
  <tr>
    <th>Data Type</th>
    <th>Data Space</th>
    <th>Data Name</th>
    <th>Data Subset</th>
  </tr>
</thead>
<tbody align="center">
  <tr>
    <td>Image Edit Generation</td>
    <td></td>
    <td>clay-v1-20240527_16_06_41</td>
    <td>default</td>
  </tr>
</tbody>
</table>


## Model Performance
Given the input "Convert this image into clay style," the following image may be generated:

![image](./image.jpg)

## Model Usage
### Command Line Execution
* Run using Scepter's SDK, taking care to use different configuration files in accordance with the different base models, as per the corresponding relationships shown below
<table>
<thead>
  <tr>
    <th rowspan="2">Base Model</th>
    <th rowspan="1">LORA</th>
    <th colspan="1">SCE</th>
    <th colspan="1">TEXT_LORA</th>
    <th colspan="1">TEXT_SCE</th>
  </tr>
</thead>
<tbody align="center">
  <tr>
    <td rowspan="8">SD1.5</td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_1.5_512_lora.yaml">lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd15_512_sce_t2i_swift.yaml">sce_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_1.5_512_text_lora.yaml">text_lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/stable_diffusion_1.5_512_text_sce.yaml">text_sce_cfg</a></td>
  </tr>
</tbody>
<tbody align="center">
  <tr>
    <td rowspan="8">SD2.1</td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_2.1_768_lora.yaml">lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd21_768_sce_t2i_swift.yaml">sce_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_2.1_768_text_lora.yaml">text_lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd21_768_text_sce_t2i_swift.yaml">text_sce_cfg</a></td>
  </tr>
</tbody>
<tbody align="center">
  <tr>
    <td rowspan="8">SDXL</td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_xl_1024_lora.yaml">lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sdxl_1024_sce_t2i_swift.yaml">sce_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_xl_1024_text_lora.yaml">text_lora_cfg</a></td>
    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sdxl_1024_text_sce_t2i_swift.yaml">text_sce_cfg</a></td>
  </tr>
</tbody>
</table>

* Running from Source Code

```shell
git clone https://github.com/modelscope/scepter.git
cd scepter
pip install -r requirements/recommended.txt
PYTHONPATH=. python scepter/tools/run_inference.py
  --pretrained_model {this model folder}
  --cfg {lora_cfg} or {sce_cfg} or {text_lora_cfg} or {text_sce_cfg}
  --prompt 'Convert this image into clay style'
  --save_folder 'inference'
```

* Running after Installing Scepter (Recommended)
```shell
pip install scepter
python -m scepter/tools/run_inference.py
  --pretrained_model {this model folder}
  --cfg {lora_cfg} or {sce_cfg} or {text_lora_cfg} or {text_sce_cfg}
  --prompt 'Convert this image into clay style'
  --save_folder 'inference'
```
### Running with Scepter Studio

```shell
pip install scepter
# Launch Scepter Studio
python -m scepter.tools.webui
```

* Refer to the following guides for model usage.

(video url)

## Model Reference
If you wish to use this model for your own purposes, please cite it as follows.
```bibtex
@misc{clay_style_edit,
    title = {clay_style_edit, {MODEL_URL}},
    author = {{USER_NAME}},
    year = {2024}
}
```
This model was trained using [Scepter Studio](https://github.com/modelscope/scepter); [Scepter](https://github.com/modelscope/scepter)
is an algorithm framework and toolbox developed by the Alibaba Tongyi Wanxiang Team. It provides a suite of tools and models for image generation, editing, fine-tuning, data processing, and more. If you find our work beneficial for your research,
please cite as follows.
```bibtex
@misc{scepter,
    title = {SCEPTER, https://github.com/modelscope/scepter},
    author = {SCEPTER},
    year = {2023}
}
```