README.md

---
license: creativeml-openrail-m
library_name: diffusers
tags:
- stable-diffusion
- stable-diffusion-diffusers
- text-to-image
- diffusers
- lora
inference: true
base_model: CompVis/stable-diffusion-v1-4
datasets:
- sylvainlapeyrade/kanji_english_meaning
language:
- en
pipeline_tag: text-to-image
---

# LoRA text2image fine-tuning - sylvainlapeyrade/kanji2english
<i>This is a model ran on only one epoch on Google Colab free version for proof of concepts purpose. Do not expect extraordinary resutls.</i>

These are LoRA adaptation weights for CompVis/stable-diffusion-v1-4. The weights were fine-tuned on the sylvainlapeyrade/kanji_english_meaning dataset to generate Kanji images based on English descriptions. This model provides a novel approach to visualizing the artistic representation of Kanji characters through a text-to-image generation process, using the powerful Stable Diffusion architecture enhanced with LoRA layers for fine-tuning.

## Model Description

This model uses LoRA (Low-Rank Adaptation) layers to fine-tune the Stable Diffusion v1-4 model specifically for the task of generating Kanji images from English text descriptions. The fine-tuning process was conducted using the sylvainlapeyrade/kanji_english_meaning dataset, which contains a collection of Kanji characters and their corresponding English meanings.

## How to Use

This model is intended to be used with the `diffusers` library. Here is an example of how to generate an image from text:

```python
from diffusers import LoraDiffusionPipeline

# Load the pipeline
pipe = LoraDiffusionPipeline.from_pretrained("sylvainlapeyrade/kanji2english")

# Generate an image
prompt = "a kanji meaning a Doge"
image = pipe(prompt).images[0]

# Save or display the image
image.save("kanji_doge.png")
```

## Limitations and Bias

This model has been trained specifically for generating Kanji characters from English text inputs with the <i>"a kanji meaning"</i> sequence and may not generalize well to other text-to-image tasks outside this scope. Additionally, the accuracy of the generated images may vary depending on the complexity and rarity of the Kanji.

## Examples

Below are some example images generated by the model:

![img_0](./kanji.jpeg)
![img_1](./kanji2.jpeg)
![img_2](./kanji3.jpeg)
![img_3](./kanji4.jpeg)

## Acknowledgements

Special thanks to the creators of the KANJIDIC and KanjiVG projects for providing the data sources used during training.

## License

This model is released under the CreativeML Open RAIL-M license, which allows for non-commercial use, sharing, and adaptation with attribution.