November PR1
Published: 2023-11-28 05:02:31

This November, I was very happy to have submitted PRs([286](https://github.com/tarasglek/chatcraft.org/pull/286),[302](https://github.com/tarasglek/chatcraft.org/pull/302)) to [ChatCraft](https://github.com/tarasglek/chatcraft.org) once again. Although it may still take some time for the PRs to be finally merged, the good news is that the basic functionality has already been implemented. I would like to record the process and the challenges I encountered.

## PR Background 
With an important update, GPT4 model now has the capability to take in images and answer questions about them. Therefore, it would be great if ChatCraft also had the capability to send images, and I am also very excited that I have the opportunity to be involved in the development of this feature.

## Develop process

### Stage 1: An easy start

Since this new feature involves many components, I first submitted an implementation version with minimal changes. Because I believe that the fewer files changed, the smaller the impact I am likely to make. Here is my first [PR](https://github.com/tarasglek/chatcraft.org/pull/281). In this PR, I only changed 3 files, but I was able to successfully call the API to [send](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uvfbyqb4e6elk04cwk9o.png) images and [obtain](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/szqpxtfvjf3xl094v46e.png) the answers to questions

However, we still have many issues that need to be resolved.
The major issues with this PR include: the images sent do not appear in the history, the messages returned are incomplete, and the user experience is not intuitive enough (I used the drag method, which is less intuitive than buttons). Professor [Humphrey](https://github.com/humphd) has provided many detailed and valuable [suggestions](https://github.com/tarasglek/chatcraft.org/pull/281#issuecomment-1805657667). This has been of significant assistance to my subsequent development. Shortly thereafter, [Taras Glek](https://github.com/tarasglek) offered me the role of Collaborator. This allowed for swift deployment of new updates. I'm deeply appreciative of this opportunity.

## Stage 2: Investigate and solve issues
In the following days, I first attempted to resolve the issue of the response message being cut off. 

![cutoff](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/plfd9ro78jxr9jehuvbh.png)

I believe it affects normal use and has a higher priority.

### The process of finding bugs

I use a python version to do the same request to see how it looks like:

![python response](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0klza6g6hwguu5vrec51.png)

I got the the details of response. I realized that it's likely the image size was too large, causing the token count to reach the threshold. Therefore, I located the type `OpenAI.Chat.ChatCompletionCreateParams`for the token limit  and set the `max_tokens` in `src/lib/ai.ts` to a particularly high value. I did this to verify my hypothesis. Consequently, I obtained the upper limit of tokens: 4096.

![token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/50kzncyjukajfmzy83bn.png)

It should be noted that this seems like it should be the default setting, as [Zima](https://community.openai.com/t/can-i-set-max-tokens-for-chatgpt-turbo/81207) said
>  "By default, the number of tokens the model can return will be (4096 - prompt tokens)."

The official [documentation](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo
) displays it as:

![doc](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3ehsi12qqpxscuy6xsm7.png)

After updating to openai@v4.20.0, I still needed to set the `max_tokens`.

## Stage 3: Refinement and Enhancement

Next came continuous improvement. In this stage, I updated more files, totaling [10](https://github.com/tarasglek/chatcraft.org/pull/286/files). 

I completed the following updates:

1 Updating the model based on file type.

I achieved this by using `setSettings` from `src/hooks/use-settings.tsx`
```tsx

  // Update model to the supported model when inputImages is not empty
  useEffect(() => {
    if (inputImages && inputImages.length > 0) {
      const modelSupportsImages = models.find((model) => model.supportsImages);
      if (modelSupportsImages) {
        setSettings({ ...settings, model: modelSupportsImages });
      }
    }
    // Assuming models and setSettings are stable and don't need to be in the dependencies array
    // eslint-disable-next-line
  }, [inputImages]);
```


2 Attaching file method.  

I added the [`ClipIcon`](https://github.com/tarasglek/chatcraft.org/pull/286/files#diff-f670f59cbbef90fc6030e478f8cd80821ad583aee3e1e172e132bdb15d5512d5) component to achieve the functionality of Attaching.

I achieved this by mimic `src/components/PromptForm/MicIcon.tsx`


3 Improved preview if the attached file is an image:
 Clicking on the small image displays a larger version with a "close" button in the middle of the window.

I achieved this by using `Modal` for display larger version.


4 Keeping images in the chat history.

I achieved this by adding `image` in [DB](https://github.com/tarasglek/chatcraft.org/pull/286/files#diff-196dc6cfdb581213173439841d9e85ada909cf40face80a3ac1b6739bbd85bf6) and [`ChatCraftMessage`](https://github.com/tarasglek/chatcraft.org/pull/286/files#diff-c5e959cfbbe3d92b50356acefecf790bea69d21cc6800bdc040d750ed68ac41d)

##Summary
This PR made updated the desktop version's UI, providing users with the ability to send images and ask questions to GPT-4. I made a series of updates based on the feedback received. The scope of this PR is noticeably larger than my PRs in October, and it has also been ongoing for a longer duration. I do indeed feel the urgency of time. I will continue to make updates until it is merged.