Speaking with LLMs locally (offline internet)
Tags: programming, ai, tutorial
Published: 2024-04-07 19:39:46

**Intro:** Even though there are contrary opinions, more and more people are hearing that LLMs (_large language models_) models are a bit intelligent. To test these AI tools, it would be interesting to know how far the limit of their knowledge goes. Until now, there are still doubts about the composition of the training dataset for some LLMs. There is a certain "mystery" in what these models are capable of answering or doing. They probably have as part of their training dataset a massive amount of digital books, files, images, audios, texts, signals, methods, and analyses. Therefore, it is expected that LLMs, including the one we used in this post, have a high potential to answer the most different questions.

<small><small>** **The cover image was generated by <a href="https://deepai.org/machine-learning-model/text2img">a text-to-image generator</a> with the choice of the phrase "A Llama fishing in a beautiful lake.**"</small></small>

**What will we do?** In this context, this post shows how to run a <u>_open source_</u> and <u>free</u> LLM model on a personal computer. To test this local version of LLM, we ask some questions that can show some of the difficulties that arise in interactions with these robots.

Maybe some coffee can help with this work! ☕

![Test 5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0sp6d8gdbqd7lyaluh5e.gif)

<center><small><small>GIF image of the Test 5, done next, in which the Mistral-7B solves a Calculus exercise.</small><small></center>

---

## Configuring the LLM chatbot (Mistral-7B)

The LLM model chosen for this post is <a href="https://huggingface.co/jartine/mistral-7b.llamafile">**`Mistral-7B`**</a>, a variation of <a href="https://llama.meta.com/llama2/">Llama 2</a>. It is available in the `llamafile` format, an executable of around 4GB that can be <a href="https://huggingface.co/jartine">downloaded from a repository</a> on the Hugging Face website.

**Llafile:** More information about the `llamafile` format can be found at this [link](https://justine.lol/cosmopolitan/index.html).

**Credits:** Both contents mentioned above are by the same <a href="https://justine.lol/">author</a> (thank you!).

### Running the Mistral-7B locally
To run this model locally, we follow the steps below.

**<u>STEP 1.</u>** <u>Download</u> the executable (`.llamafile`) by clicking <a href="https://huggingface.co/jartine/mistral-7b.llamafile">here</a>.

![Image of Hugging Face download page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/910x4uqxzrv6n3ovl6ag.png)

**Notes:**
- The <u>name of the downloaded file</u> in this tutorial is `mistral-7b-instruct-v0.1-Q4_K_M.llamafile`. For other versions of the Mistral, the name will probably be different.
- After downloading the `llamafile` file, we can turn off our Internet connection.

**<u>STEP 2.</u>** <u>Change the permissions</u> of the downloaded file `.llamafile` so that we can run it on our computer.

On Linux, this can be done using the command below:
```shell
chmod 755 mistral-7b-instruct-v0.1-Q4_K_M.llamafile
```
In Windows, by right-clicking on the file you can change yours permissions.

**<u>STEP 3.</u>** <u>Run</u> the downloaded file. On Linux this can be done with
```shell
./llava-v1.5-7b-q4-server.llamafile
```
In Windows, we run the file by double-clicking on it.

**<u>STEP 4.</u>** Now just <u>browse</u> the Mistral-7B by accessing port 8080 of the localhost [http://127.0.0.1:8080](http://127.0.0.1:8080/), which should automatically open in the default web browser of the computer used after completing the previous step.

---

## Local tests with the Mistral-7B

In order to test the Mistral-7B model, which is now running locally, we will ask you to answer some questions. For testing purposes, we will ask it to change the level of understanding expected in his answers. Skills like this will make our AI chatbot a robot that not only reads what is in a dictionary, encyclopedia, or database but is also capable of transmitting a message from the sending source to a receiving source, according to the requirements and characteristics of each of these.

Below are the questions and answers provided in the tests.

- **Test 1.** _"Solve this equation for x: x^2 - 2*x + 3 = 0."_ (<u>Expected answer:</u> x = 1, x = 3)

<small>- **Answer 1 (summarized):** _"To solve we can use factoring to rewrite the equation as (x-1)(x-3). This gives us two possibles solutions: x = 1 or x = 3."_</small>

![Test 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xj8w3kak1ii49px9euri.gif)

- **Test 2.** _"Explain to an <u>elementary school student</u> how the `for` repetition loop works in the Python language. Present a teaching example at this level of understanding."_

<small>- **Answer 2 (summarized):** _"The `for` loop is used when we want to repeat a block or section of code until a certain condition is reached. As an example, consider a list `numbers = [1, 2, 3, 4, 5]` , so to `print` `Hello!` 5 times is done like this:_
```python
for n in numbers:
   print(n)
```
</small>

![Test 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n2bglcgzx35knzql7d92.gif)

- **Test 3.** _"Explain to a <u>graduate student</u> in the area of Computing, how the `for` repetition loop works in the Python language. Present a didactic example. Present a didactic example at this level of understanding."_

   - **Answer 3:** _"Answers practically the same as in Test 2, but uses language with more technical terms and presents a simpler example than the previous one."_

![Test 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xcoa2mhwxc381fxe30az.gif)

- **Test 4.** _"Please, could you tell a joke in Brazilian Portuguese?"_

  - **Answer 4:**


![Test 4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kb0iiub98agapucvz7v0.gif)
- **Test 5.** _"If_ `x=sum_{i=3}^{102}(i-2)`_, what is the value of_ `x`_?"_ (<u>Expected answer:</u> `x = 5050`)

  - **Answer 5:**

![Test 5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8dd91xpkg1culm3wfbpr.gif)

**Comments on answers:**

- **Tests 1.** Nailed it!

- **Tests 2 and 3.** When asked to change the complexity of understanding their answers, they understand and, in a way, follow.

- **Test 4.** Maybe if we had written it in English instead of Portuguese, or changed its hyperparameters, it would have been funnier.

- **Test 5.** He got the answer wrong, but when we changed it to the error, he knew how to correct himself and justify his mistake. Not bad for a chatbot running locally, with limited resources. From an ethical point of view, it turned out to be a somewhat friendly robot.
  
---

## License

Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

---

## Some next steps

Another option for increment this post, is test other free model that can run locally. Some options are shared in this <a href="https://www.makeuseof.com/the-best-local-offline-llms-you-can-try-right-now/">link</a>.

**Can an LLM run on any hardware?** With few changes to what we did here, it is possible to create other </u>free experiments locally</u>. With around 6GB of RAM is it possible to play a "little". See more about clicking <a href="https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile">here</a>.

**Fun fact:** Although some LLM models seem <u>neutral</u> in their answers, until now, it is impossible to know really <u>how much these models know</u>, how much <u>they don't know </u> and how much <u>unprecedented knowledge</u> they can generate. For example, in the article <a href="https://www.nature.com/articles/s42256-023-00775-6">_"Generation of 3D molecules in pockets via a language model"_</a> the authors shown how LLM can helps generate scientific knowledge with novel content in the literature. Namely, they generated <a href="https://en.wikipedia.org/wiki/Chemical_synthesis">synthesizable</a> 3D molecular structures.

---

If you try the post, kindly share the results with us. Suggestions, criticisms, and opinions are welcome.