Part 4 - Page Processor
Published: 2023-10-08 15:00:51

## Introduction

In this part of the series, we will discuss the Page Processor ([link](https://github.com/hatemfaheem/ai-story-generator/blob/main/processors/page_processor.py)). Below are 2 examples from a story called "The Dark Forest", that raw input of the page processor on the left and the expected output on the right. And as you can see, one time we put the text right and another time we put it on the left. This is a simple alternating logic that we will discuss here as well.


![Page0](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5dzfwclo09m5fbznopbc.png)


![Page11](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/shpf9ns0scjku5be80ez.png)

## Text Image

Let's start with writing text to an image. Here's a simple method that will draw white text on black background image with the default font.

```python
def text_image():
    # input parameters
    text_message = "This is the fantastic story line"
    width, height = 256, 64    # image size
    bg_color = (0, 0, 0)    # black color
    font_color = (256, 256, 256)    # white color  
    
    # create an empty image with width, height and background color
    image = Image.new('RGB', (width, height), bg_color)

    # draw text on image
    draw = ImageDraw.Draw(image)
    draw.text((10, 30), text_message, fill=font_color)

    return image
```

This would result in the following

![simple text image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wpk66j25ojosmvn6zqd6.jpg)

Now let's improve it by using a custom font. I used a font called `Playfulist` and you can find it [here](https://github.com/hatemfaheem/ai-story-generator/blob/main/fonts/Playfulist.ttf). We need to define a font size, use to define a font object and pass the font object to `draw.text`.

```python
...
font_size = 12
font = ImageFont.truetype("Playfulist.ttf", font_size)
...
draw.text(
    (10, 30),
    text_message,
    font=font,
    fill=font_color,
)
```

![Custom font](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/itmnvbcdju2ajct5k0pf.jpg)

Now, instead of hard-coding (10, 30) as the text location, which is the top left corner of the text, let's put it in the center of the image. To do so, we need first to calculate the size of the text in pixels. Hence, we will use `ImageDraw.textbbox` which will return a bounding box (in pixels) of given text relative to given anchor when rendered in font with provided direction, features, and language ([reference](https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html#PIL.ImageDraw.ImageDraw.textbbox)).

We can call `draw.textbbox` then `draw.rectangle`. to visualize it.

```python
x, y, w, h = draw.textbbox((0, 0), text_message, font=font)
draw.rectangle((x, y, w, h), outline=(0, 255, 0), width=1)
```

![bbox](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x1atrz9qsihtpre7iv0f.jpg)

To shift that box to the center of the image we need a little bit of geometry. We know the center point of the image lies at `x = width/2` and `y = height/2`. Let's highlight it with blue color.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8a4e9s0lb33s5h8suc72.jpg)

Now, we want the green box to be centered around the blue dot exactly. But when we render text, we just need the top left corner. So, given our knowledge of the overall textbox size, we need to calculate the top left corner in respect to the center point. How do we do that? We start at the center point and we move left half of the width of the textbox and we move up half of the height of the text box.

```python
top_left_x = center_x - (bbox_width / 2)
top_left_y = center_y - (bbox_height / 2)

# center point of the image
center_x = width / 2
center_y = height / 2

# from the bounding box results from 'draw.textbbox'
bbox_width = w - x
bbox_height = h - y

"""
But we also know that we started from 0, 0 when
calculating the bounding box. We can just ignore x, y.
Although, when we look at the actual values in the above example, we got x = 0 and y = 3. That's probably
because PIL adds some margin for specific fonts.
But we can ignore the tiny little margins to simplify
the calculation.
"""

# assuming x = 0 and y = 0
bbox_width = w
bbox_height = h

# Hence
top_left_x = (width / 2) - (w / 2)
top_left_y = (height / 2) - (h / 2)

# And we can simplify it more to 
top_left_x = (width - w) / 2
top_left_y = (height - h) / 2
```

Let's draw a rectangle with these calculations first by using:

```python
draw.rectangle(((width - w) / 2, (height - h) / 2, (width - w) / 2 + w, (height - h) / 2 + h), outline=(255, 0, 0), width=1)
```

![center rectangle](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o80q1u94apxgz93rcik5.jpg)

Then, let's add back the text. And as you can see below, it fits the expected rectangle exactly.

```python
draw.text(
    ((width - w) / 2, (height - h) / 2),
    text_message,
    font=font,
    fill=font_color,
)
```

![text in center](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/61fw6jbgcn3k0uv4gadz.jpg)

Removing the guide lines, we get the following nicely centered text.

![nicely centered](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4uefnq4gyfn9oguznw9b.jpg)

## Justify Text

To break the story sentences into multiple lines to look better on the page, I chose a library called [JustifyText](https://pypi.org/project/JustifyText/). All we need to do is give it the original sentence and the character width, and it will break it down into smaller sentences and return them as an list of strings.

```python
text_message = "Once upon a time, there was a small village nestled deep in the heart of the Dark Forest."
results = justify(text_message, 20)
for r in results:
    print(r)
```

Running this will result the following same length strings.

```python
Once  upon  a  time,
there  was  a  small
village nestled deep
in the heart of  the
Dark Forest.
```

And with exact same image calculations we did before, we get this.

![multiline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6uppc41syuaa66fza6c7.jpg)

## Background Color

We are almost done with the text part of the image. As a last step what if we can set the background color of the text part to be somehow relevant to the image part of the page. To do this, we can calculate the dominant color of the image, lighten it to always keep a light background (that's a personal preference) and set the background color of the text part to that calculated color.

We can find a dominant color algorithm on the internet and try to implement it, but like lots of other things in this world, there's already a library that can do this for us. It's called [fast-colorthief](https://pypi.org/project/fast-colorthief/). We will just need to convert the image to numpy array before calling `get_dominant_color`.

```python
from fast_colorthief import get_dominant_color

rgba_image = story_page_content.image.convert("RGBA")
ndarray = numpy.array(rgba_image).astype(numpy.uint8)
dominant_color = get_dominant_color(ndarray, quality=1)
```

And to lighten the color we can use the following method. The idea behind this method is that we want to move the color to the white side of the color spectrum with a factor called `_BACKGROUND_TINT_FACTOR`, on the 3 RGB values. For example, given a single value between `0` and `255` that's `150`, to make it need to move it towards `255`. Hence, we need to add another value to it. How much do we add, in the below method, I calculated the difference between `255` and the current value e.g. `255 - 150` that's `105`, then multiply that by the tint factor e.g. `0.7`. That gives us `73`, which we then add to `150` to get `223`. So, we moved it towards 255 by 70% factor. In order for this to make more sense, think about the extreme cases, a tint factor of `1.0` would give us `255` for any value, that's white, 100% lightened. A factor of `0.0` would give us the original value, which means it's lightened by 0%.

```python
def _lighten_color(self, color: Tuple[int, int, int]) -> Tuple[int, int, int]:
    return (
        int(color[0] + (255 - color[0]) * self._BACKGROUND_TINT_FACTOR),
        int(color[1] + (255 - color[1]) * self._BACKGROUND_TINT_FACTOR),
        int(color[2] + (255 - color[2]) * self._BACKGROUND_TINT_FACTOR),
    )
```

The below image shows, the generated image on the left, the calculated dominant color in the middle and the lighted version of the right. Now, we can make that the background color for the text part of the image to make it better blend with the generated image in the same page.

![dominant](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nzwoyq7kcfzpg2xououw.jpg)

Going back to text image section, you can easily see we can just change the input parameters for background color and font color to lightened version of dominant color and black (or any other color) respectively. Here's an example:

![colored back](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/68tvd1qz9j9xcs6xwnok.jpg)

**Note: to make the background color of the text image even cooler, I added a gradient effect that you can find [here](https://github.com/hatemfaheem/ai-story-generator/blob/main/util/gradient_image.py). I'll skip it in this tutorial but you can take a look if you are interested.**

## Put them together

Now that we have the generated image and a nice looking text part, let's put them together. One on the left side and one on the right side. To do this the 2 images must be of the same height. We can create an empty image pf the expected size and `paste` each part on the correct location.

```python
result = Image.new(
    "RGB", (image_left.width + image_right.width, image_left.height)
)
result.paste(image_left, (0, 0))
result.paste(image_right, (image_left.width, 0))
```

And to make sure that it looks and feels like a book, we can alternate between text on left and text on right every other page.

```python
if int(story_page_content.page_number) % 2 == 0:
    page_image: Image.Image = self._concat_horizontally(
        story_page_content.image, text_img
    )
else:
    page_image = self._concat_horizontally(text_img, story_page_content.image)
```

## Paper Wrinkling Effect

That's actually a nice simple technique that you can use to add any other effect to the whole page. It blends an existing page image ([here](https://github.com/hatemfaheem/ai-story-generator/blob/main/images/paper.jpeg)) with the full page with a factor. Using `Image.blend` from PIL.

```python
def _add_paper_effect(self, page_image: Image) -> Image:
    paper: Image = Image.open(self._PAPER_IMAGE_PATH).convert(page_image.mode)
    paper = paper.resize(page_image.size)
    return Image.blend(page_image, paper, self._PAPER_BLEND_FACTOR)
```

## Bring it all together

To bring it all together we need to do the following steps:

1. Calculate dominant/lightened background color.
2. Create text image
3. Concatenate image and text parts
4. Add paper wrinkling effect
5. Save and return

```python
def create_page(
    self,
    workdir: str,
    story_page_content: StoryPageContent,
    audio: AudioInfo,
    story_size: StorySize,
) -> StoryPage:

    # calculate dominant/lightened background color
    background_color = self._calculate_background_color(story_page_content)

    # create text part
    text_img: Image.Image = self._create_text_image(
        size=(story_size.text_part_width, story_size.text_part_height),
        bg_color=background_color,
        message=story_page_content.sentence,
        font=ImageFont.truetype(self._FONT, story_size.font_size),
        font_color=self._BLACK_COLOR,
    )

    # alternate between text on left and text on right
    if int(story_page_content.page_number) % 2 == 0:
        page_image: Image.Image = self._concat_horizontally(
            story_page_content.image, text_img
        )
    else:
        page_image = self._concat_horizontally(text_img, story_page_content.image)

    # add paper effect
    page_image = self._add_paper_effect(page_image)
    page_filepath = os.path.join(
        workdir, f"page_{story_page_content.page_number}.png"
    )

    # save and return
    page_image.save(page_filepath)
    return StoryPage(
        page_content=story_page_content,
        page_image=page_image,
        page_filepath=page_filepath,
        audio=audio,
    )
```

## Next

In this section, we did lots of image processing, we played with colors and fonts, we did a little bit of geometry to correctly place the text in the center. We can also keep adding endless effects to how the page look and feel. And there are also lots of parameters we can play with. This includes, font size and color, tinting factors for background color and paper effect, etc.

Now that we have the nice looking pages built in this section and the our AI narrator ready from previous section. Let's compile the final video in the next part.