Introducing LM Studio
Tags: localai, huggingface, lmstudio, llm
Published: 2023-11-12 21:00:56

This brief article presents [LM Studio](https://lmstudio.ai), a handy tool for installing and testing open source LLMs on your desktop. LM Studio, which is not open source, utilizes the popular [llama.cpp](https://github.com/ggerganov/llama.cpp) library. Be aware that the [terms of use](https://lmstudio.ai/terms) prohibit modifications, redistribution, and commercial use of LM Studio. Additionally, users are given a 30-day notice period before any changes to these terms take effect.

![LM Studio Logo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g4db066l5wpqoo6gw63a.png)

LM Studio is user-friendly and available in binary format for Windows and Mac, with a Linux version in the works. It supports various models compatible with the [ggml](https://github.com/ggerganov/ggml) tensor library from the llama.cpp project and requires 16 GB of RAM.

Let's explore how LM Studio is both easy to use and convenient.

## Using Models from the Chat panel

After installation, LM Studio facilitates the downloading of models from the Hugging Face Hub, including preset options.

For example, we can download the [Zephyr 7B β](https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF) model, adapted by TheBloke for llama.cpp's GGUF format.

![Downloading a Model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ajpqvn5o308pfcjsapn1.png)

Activating and loading the model into LM Studio is straightforward.

![Loading a Model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ba0v1sqmhnxzb666bd82.png)

You can then immediately start using the model from the Chat panel, no Internet connection required.

![Chatting with Zephyr](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rigfgjv33egol0eooaiq.png)

The right panel displays and allows modification of default presets for the model. Memory usage and useful inference metrics are shown in the window's title and below the Chat panel, respectively.

Other models, like [codellama Instruct 7B](https://huggingface.co/TheBloke/CodeLlama-7B-Instruct-GGUF), are also available for download and use.

![Using Codellama](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nkoz03gup8478w2rjxgs.png)

LM Studio also highlights new models and versions from Hugging Face, making it an invaluable tool for discovering and testing the latest releases.

## Accessing Models with APIs

A notable feature of LM Studio is the ability to create Local Inference Servers with just a click.

![Local Inference Server](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ixvfoawzsbor9yvf5zf4.png)

The *Automatic Prompt Formatting* option simplifies prompt construction to match the model's expected format. The exposed API aligns with the OpenAI format.

Here's an example of calling the endpoint with CURL:

```sh
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{ 
  "messages": [ 
    { "role": "system", "content": "You are an AI assistant answering Tech questions" },
    { "role": "user", "content": "What is Java?" }
  ], 
  "temperature": 0.7, 
  "max_tokens": -1,
  "stream": false
}'
```

The response provides the requested information:

```json
{
    "id": "chatcmpl-iyvpdtqs1qzlv6jqkmdt9",
    "object": "chat.completion",
    "created": 1699806651,
    "model": "~/.cache/lm-studio/models/TheBloke/zephyr-7B-beta-GGUF/zephyr-7b-beta.Q4_K_S.gguf",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Java is a high-level, object-oriented
								programming language that was first released by Sun
								Microsystems in 1995. It is now owned by Oracle Corporation.
								Java is designed to be platform independent, meaning that it
								can run on any operating system that has a Java Virtual
								Machine (JVM) installed. Java's primary applications are in
								the development of desktop applications, web applications,
								and mobile apps using frameworks such as Android Studio,
								Spring Boot, and Apache Struts. Its syntax is similar to
								C++, but with added features for object-oriented programming
								and memory management that make it easier to learn and use
								than C++. Java's popularity is due in part to its extensive
								library of pre-written code (known as the Java Class
								Library) which makes development faster and more efficient."
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 0,
        "completion_tokens": 166,
        "total_tokens": 166
    }
}
```

This feature greatly aids in testing integrations with frontends like chatbots or workflow solutions like [Flowise](https://dev.to/worldlinetech/enhance-your-website-with-ai-embed-a-gpt-chatbot-with-flowise-jd6).

## Conclusion

Although not open source, LM Studio is a robust addition to your local toolkit, allowing you to easily experiment with and adopt models from Hugging Face. Its user-friendly interface and versatile features make it an essential resource for anyone looking to delve into the world of large language models.

## Video

You can also check the video tutorial from my [GenAI's Lamp](https://www.youtube.com/playlist?list=PLpEPN3tmJMNaQg4spmz1y2kGcqFaE2_8m) YouTube series.

{% embed https://www.youtube.com/watch?v=NLSmAeS9iZM %}