Creating Style Previews with Stable Diffusion
Tags: ai, webdev, learning
Published: 2023-09-20 02:29:35

# Enhancing Your Shopping Experience with AI and LORA

While wisiting a cloating e-commerce shop, I had a random thought: "Wouldn't it be cool to preview clothes on myself while shopping?" As a developer by trade, and having prior experience with stable diffusion using Dream Booth, I decided to build an MVP to do just that.

## Exploring LORA

First, I attempted to use Dream Booth to learn my face. However, I stumbled upon something even cooler: LORA (Low-Rank Adaptation). LORA proved to be faster and easier to train and had the potential for combining different aspects. For instance, you could combine a LORA model of a user's face with a LORA model of the clothes you want that user to wear.

I started by creating a LORA model of my face and had some fun with it, generating cool images like this one:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/13yngjmppey459o6itm3.png)

Then, I created a LORA model of jackets from images found on an e-commerce website:

![Jacket images](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/akuf45bgkpf5sqegx5wb.png)

Next, I combined these LORA models in a prompt and achieved pretty decent results:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o6wniv79om1d4szo067t.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2m7vkx6i9yzb4n1bcvj9.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pudlr3ol2ns9mzgjn2q.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mot9f7d51hwbbxwm5uc2.png)

However, I noticed that the images weren't perfect, especially when it came to small details of the jacket and my face. I ran more prompts and tried to improve the LORA model, but my face was consistently off.

## Troubleshooting the LORA Model

I experimented with several solutions:

1. I attempted to label a person in the jacket dataset as a model and included this as a negative prompt, but it didn't yield significant improvements.

2. I refined the prompts and used WD4 to annotate images, but the results remained suboptimal.

After trying numerous approaches without groundbreaking success, I decided to explore using a control net to correct the person's pose and apply masks to allow LORA to work on each part of the image independently. Here's what it looked like:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5it8vq8aalre3uhvf8cb.png)

The results were better, though my face was still not quite right. At first glance, everything seemed to be working okay:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44sieav92mf5jbaxnbp4.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yu0l9rnl8nbt1fch4pum.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hlqquathm4xdxh53xobk.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6xnkeio5jov9t8jbomxk.png)

Encouraged by these results, I attempted to add another piece of clothing to see if it would work with three LORA models. Here are the original images from the shop:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4lklbr7odgja9oewj15k.png)

And the results:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6h80urqns9yrusw3ad3b.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/igwau62ss4jdqtrwk3pz.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mghijes0mhir69gktti6.png)

This time, the jacket and jeans looked really good, but the face was worse than in the last batch. Unfortunately, I couldn't pinpoint the exact reason for this.

I'm still working on solving this issue, and when I do, I hope to share it with the community. If anyone has any insights or solutions, I would be incredibly grateful.