Building a Short Story Assistant using Cloudflare AI
Tags: cloudflarechallenge, devchallenge, ai
Published: 2024-04-08 16:55:37

<!-- Don't forget to add a cover image or an additional tag to this post (if you want). -->

*This is a submission for the [Cloudflare AI Challenge](https://dev.to/devteam/join-us-for-the-cloudflare-ai-challenge-3000-in-prizes-5f99).*

## What I Built

Short Stories Assistant is a simple tool that helps writers generate short stories of about 2000 words using Text Generation models. 

An example of story generation flow would be:
- Providing a prompt using an audio file (optional) ;
- Generation a story using one of the available models (e.g. Mistral 7b) ;
- Editing and saving the story (this will automatically generate the story title, summary, and cover image) ;
- Improving the story title and summary ;
- View previously generated stories.

The application acts as a human assistant, and, therefore, allows the user to customize the story, the title, and the summary at any step of the flow.

## Demo

The application is live at https://demo.short-stories-assistant.pages.dev/


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pwhijs9vfi4yajzjfv2.png)

## My Code

Code available at https://github.com/jonathanarnault/short-stories-assistant

Technologies used:
- [Remix](https://remix.run/) ;
- [TailwindCSS](https://tailwindcss.com/) and [DaisyUI](https://daisyui.com/) ;
- Cloudflare Pages ;
- Cloudflare Workers AI ;
- IndexedDB.

## Journey

I never used Cloudflare workers before this challenge, but I was looking to test it on a side project since it seemed to be a powerful IaaS. This challenge was an opportunity to test the service on a simple project.

Since this challenge was focused on Workers AI, I wanted to build a Proof of Concept with basic features for a Short Story Assistant. Thus, I decided to only store data client-side and focus on using as many task types as possible that can fit in the app.

For the Short Story Assistant, I used the following task types:
- Speech Recognition (whisper), for prompting the service using audio files ;
- Text Generation (mistral-7b-instruct-v0.1, falcon-7b-instruct, llama-2-13b-chat-awq), for the story generation ;
- Text Summarization (bart-large-cnn) for the story abstract and title ;
- Text to Image (dreamshaper-8-lcm) for the story cover image generation.

During the development, I was impressed by the ease of integration of Workers AI inside a project. I was able in a few hours to implement a fully functional Proof of Concept with many task types. The Cloudflare SDK makes it easy to call an AI service and get results back.

Despite the ease of integration, I faced some technical challenges with the AI services integration:
- First of all, when I tried to implement text generation, the service usually timed out with partial text. I solved this issue using an EventSource to stream data and avoid timeout ;
- The second challenge that I experienced was to find a text generation model that does not ignore my prompts while following system prompts. While Mistral 7b was working well with my prompt, I had to skip Llama 2 7b and Phi-2 as they were not answering my prompts correctly ;
- Finally, while the Remix integration with Cloudflare Pages is great, especially the deployment, I did not find an easy way to call Workers AI in dev mode. To solve this, I simply ran the application in preview mode locally to test the integration.

In conclusion, this was a fun project to implement and I recommend using Cloudflare Workers AI in projects that do not need custom models. I will also probably test more features of Cloudflare Workers in the future as they provide a great set of services to build web applications.