init

README.md

@@ -0,0 +1,93 @@
+---
+frameworks:
+- Pytorch
+license: apache-2.0
+tasks:
+- image-style-transfer
+
+#model-type:
+##如 gpt、phi、llama、chatglm、baichuan 等
+#- gpt
+
+#domain:
+##如 nlp、cv、audio、multi-modal
+#- nlp
+
+#language:
+##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
+#- cn 
+
+#metrics:
+##如 CIDEr、Blue、ROUGE 等
+#- CIDEr
+
+#tags:
+##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
+#- pretrained
+
+#tools:
+##如 vllm、fastchat、llamacpp、AdaSeq 等
+#- vllm
+---
+# StyleBooth: Image Style Editing with Multimodal Instruction
+
+## Run StyleBooth
+- Code implementation: See model configuration and code based on [🪄SCEPTER](https://github.com/modelscope/scepter).
+
+- Demo: Try [🖥️SCEPTER Studio](https://github.com/modelscope/scepter/tree/main?tab=readme-ov-file#%EF%B8%8F-scepter-studio). 
+
+- Easy run:
+Try the following example script to run StyleBooth modified from [tests/modules/test_diffusion_inference.py](https://github.com/modelscope/scepter/blob/main/tests/modules/test_diffusion_inference.py):
+
+```python
+# `pip install scepter>0.0.4` or
+# clone newest SCEPTER and run `PYTHONPATH=./ python <this_script>` at the main branch root.
+import os
+import unittest
+
+from PIL import Image
+from torchvision.utils import save_image
+
+from scepter.modules.inference.stylebooth_inference import StyleboothInference
+from scepter.modules.utils.config import Config
+from scepter.modules.utils.file_system import FS
+from scepter.modules.utils.logger import get_logger
+
+
+class DiffusionInferenceTest(unittest.TestCase):
+    def setUp(self):
+        print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))
+        self.logger = get_logger(name='scepter')
+        config_file = 'scepter/methods/studio/scepter_ui.yaml'
+        cfg = Config(cfg_file=config_file)
+        if 'FILE_SYSTEM' in cfg:
+            for fs_info in cfg['FILE_SYSTEM']:
+                FS.init_fs_client(fs_info)
+        self.tmp_dir = './cache/save_data/diffusion_inference'
+        if not os.path.exists(self.tmp_dir):
+            os.makedirs(self.tmp_dir)
+
+    def tearDown(self):
+        super().tearDown()
+
+    # uncomment this line to skip this module.
+    # @unittest.skip('')
+    def test_stylebooth(self):
+        config_file = 'scepter/methods/studio/inference/edit/stylebooth_tb_pro.yaml'
+        cfg = Config(cfg_file=config_file)
+        diff_infer = StyleboothInference(logger=self.logger)
+        diff_infer.init_from_cfg(cfg)
+
+        output = diff_infer({'prompt': 'Let this image be in the style of sai-lowpoly'},
+                            style_edit_image=Image.open('asset/images/inpainting_text_ref/ex4_scene_im.jpg'),
+                            style_guide_scale_text=7.5,
+                            style_guide_scale_image=1.5,
+                            stylebooth_state=True)
+        save_path = os.path.join(self.tmp_dir,
+                                 'stylebooth_test_lowpoly_cute_dog.png')
+        save_image(output['images'], save_path)
+
+
+if __name__ == '__main__':
+    unittest.main()
+```

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"image-style-transfer"}

datasets/stylebooth_dataset.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:888263d7c24de3b4000ba8714d74e2051ce2b2e88dc593786478fd12441d2204
+size 3273029877

models/stylebooth-tb-5000-0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a89eba48e77030f312f1834de44acfe8fc64a452f4f61d05776b45a18f530ae
+size 4265309292

tuners/clay_style_edit/0_SwiftLoRA/adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": null,
+  "bias": "none",
+  "enable_lora": null,
+  "fan_in_fan_out": false,
+  "inference_mode": false,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.0,
+  "lora_dtype": null,
+  "lr_ratio": 16.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "model_key_mapping": null,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "swift_type": "LORA",
+  "target_modules": "model.*(to_q|to_k|to_v|to_out.0|net.0.proj|net.2)$",
+  "task_type": null,
+  "use_dora": false,
+  "use_merged_linear": false,
+  "use_qa_lora": false,
+  "use_rslora": false
+}

tuners/clay_style_edit/0_SwiftLoRA/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c929890675f2463b7120dc27d2516627cde5d6dd16588f13a3fb1fbd851e6ee
+size 383114637

tuners/clay_style_edit/README.md

@@ -0,0 +1,168 @@
+---
+frameworks:
+- Pytorch
+license: apache-2.0
+tasks:
+- efficient-diffusion-tuning
+---
+
+<p align="center">
+
+  <h2 align="center">clay_style_edit</h2>
+  <p align="center">
+    <br>
+        <a href="https://github.com/modelscope/scepter/"><img src="https://img.shields.io/badge/powered by-scepter-6FEBB9.svg"></a>
+    <br>
+  </p>
+
+## Model Introduction
+Transfer images into clay style
+
+## Model Parameters
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Base Model</th>
+    <th rowspan="2">Tuner Type</th>
+    <th colspan="4">Training Parameters</th>
+  </tr>
+  <tr>
+    <th>Batch Size</th>
+    <th>Epochs</th>
+    <th>Learning Rate</th>
+    <th>Resolution</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td rowspan="8">EDIT</td>
+    <td>LORA</td>
+    <td>1</td>
+    <td>50</td>
+    <td>0.0001</td>
+    <td>[512, 512]</td>
+  </tr>
+</tbody>
+</table>
+
+
+<table>
+<thead>
+  <tr>
+    <th>Data Type</th>
+    <th>Data Space</th>
+    <th>Data Name</th>
+    <th>Data Subset</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td>Image Edit Generation</td>
+    <td></td>
+    <td>clay-v1-20240527_16_06_41</td>
+    <td>default</td>
+  </tr>
+</tbody>
+</table>
+
+
+## Model Performance
+Given the input "Convert this image into clay style," the following image may be generated:
+
+![image](./image.jpg)
+
+## Model Usage
+### Command Line Execution
+* Run using Scepter's SDK, taking care to use different configuration files in accordance with the different base models, as per the corresponding relationships shown below
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Base Model</th>
+    <th rowspan="1">LORA</th>
+    <th colspan="1">SCE</th>
+    <th colspan="1">TEXT_LORA</th>
+    <th colspan="1">TEXT_SCE</th>
+  </tr>
+</thead>
+<tbody align="center">
+  <tr>
+    <td rowspan="8">SD1.5</td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_1.5_512_lora.yaml">lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd15_512_sce_t2i_swift.yaml">sce_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_1.5_512_text_lora.yaml">text_lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/stable_diffusion_1.5_512_text_sce.yaml">text_sce_cfg</a></td>
+  </tr>
+</tbody>
+<tbody align="center">
+  <tr>
+    <td rowspan="8">SD2.1</td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_2.1_768_lora.yaml">lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd21_768_sce_t2i_swift.yaml">sce_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_2.1_768_text_lora.yaml">text_lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sd21_768_text_sce_t2i_swift.yaml">text_sce_cfg</a></td>
+  </tr>
+</tbody>
+<tbody align="center">
+  <tr>
+    <td rowspan="8">SDXL</td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_xl_1024_lora.yaml">lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sdxl_1024_sce_t2i_swift.yaml">sce_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/examples/generation/stable_diffusion_xl_1024_text_lora.yaml">text_lora_cfg</a></td>
+    <td><a href="https://github.com/modelscope/scepter/blob/main/scepter/methods/scedit/t2i/sdxl_1024_text_sce_t2i_swift.yaml">text_sce_cfg</a></td>
+  </tr>
+</tbody>
+</table>
+
+* Running from Source Code
+
+```shell
+git clone https://github.com/modelscope/scepter.git
+cd scepter
+pip install -r requirements/recommended.txt
+PYTHONPATH=. python scepter/tools/run_inference.py
+  --pretrained_model {this model folder}
+  --cfg {lora_cfg} or {sce_cfg} or {text_lora_cfg} or {text_sce_cfg}
+  --prompt 'Convert this image into clay style'
+  --save_folder 'inference'
+```
+
+* Running after Installing Scepter (Recommended)
+```shell
+pip install scepter
+python -m scepter/tools/run_inference.py
+  --pretrained_model {this model folder}
+  --cfg {lora_cfg} or {sce_cfg} or {text_lora_cfg} or {text_sce_cfg}
+  --prompt 'Convert this image into clay style'
+  --save_folder 'inference'
+```
+### Running with Scepter Studio
+
+```shell
+pip install scepter
+# Launch Scepter Studio
+python -m scepter.tools.webui
+```
+
+* Refer to the following guides for model usage.
+
+(video url)
+
+## Model Reference
+If you wish to use this model for your own purposes, please cite it as follows.
+```bibtex
+@misc{clay_style_edit,
+    title = {clay_style_edit, {MODEL_URL}},
+    author = {{USER_NAME}},
+    year = {2024}
+}
+```
+This model was trained using [Scepter Studio](https://github.com/modelscope/scepter); [Scepter](https://github.com/modelscope/scepter)
+is an algorithm framework and toolbox developed by the Alibaba Tongyi Wanxiang Team. It provides a suite of tools and models for image generation, editing, fine-tuning, data processing, and more. If you find our work beneficial for your research,
+please cite as follows.
+```bibtex
+@misc{scepter,
+    title = {SCEPTER, https://github.com/modelscope/scepter},
+    author = {SCEPTER},
+    year = {2023}
+}
+```

tuners/clay_style_edit/configuration.json

@@ -0,0 +1 @@
+{}

tuners/clay_style_edit/params.yaml

@@ -0,0 +1,32 @@
+DESCRIPTION: Transfer images into clay style
+PARAMS:
+  base_model: edit
+  base_model_revision: EDIT
+  bucket_no_upscale: false
+  bucket_resolution_steps: 64.0
+  data_source: Dataset Management
+  data_type: Image Edit Generation
+  enable_resolution_bucket: false
+  eval_prompts: Convert this image into clay style
+  learning_rate: 0.0001
+  lora_alpha: 256.0
+  lora_rank: 256.0
+  max_bucket_resolution: 1024.0
+  min_bucket_resolution: 256.0
+  ms_data_space: ''
+  ms_data_subname: default
+  ori_data_name: clay-v1-20240527_16_06_41
+  prompt_prefix: ''
+  push_to_hub: false
+  replace_keywords: ''
+  resolution_height: 512
+  resolution_width: 512
+  save_interval: 25
+  sce_ratio: 1
+  text_lora_alpha: 256.0
+  text_lora_rank: 256.0
+  train_batch_size: 1
+  train_epoch: 50
+  tuner_name: LORA
+  work_dir: ''
+  work_name: ''