Spaces:
Paused
Paused
File size: 4,663 Bytes
8412920 68bb241 8412920 a347719 526ff2e 11c65a0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 |
---
title: LLMServer
emoji: 👹
colorFrom: indigo
colorTo: purple
sdk: docker
pinned: false
---
# LLM Server
This repository contains a FastAPI-based server that serves open-source Large Language Models from Hugging Face.
## Getting Started
These instructions will help you set up and run the project on your local machine.
### Prerequisites
- Python 3.10 or higher
- Git
### Cloning the Repository
Choose one of the following methods to clone the repository:
#### HTTPS
```bash
git clone https://huggingface.co/spaces/TeamGenKI/LLMServer
cd project-name
```
#### SSH
```bash
git clone git@hf.co:spaces/TeamGenKI/LLMServer
cd project-name
```
### Setting Up the Virtual Environment
#### Windows
```bash
# Create virtual environment
python -m venv myenv
# Activate virtual environment
myenv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
```
#### Linux
```bash
# Create virtual environment
python -m venv myenv
# Activate virtual environment
source myenv/bin/activate
# Install dependencies
pip install -r requirements.txt
```
#### macOS
```bash
# Create virtual environment
python3 -m venv myenv
# Activate virtual environment
source myenv/bin/activate
# Install dependencies
pip3 install -r requirements.txt
```
### Running the Application
Once you have set up your environment and installed the dependencies, you can start the FastAPI application:
```bash
uvicorn main.app:app --reload
```
The API will be available at `http://localhost:8001`
### API Documentation
Once the application is running, you can access:
- Interactive API documentation (Swagger UI) at `http://localhost:8000/docs`
- Alternative API documentation (ReDoc) at `http://localhost:8000/redoc`
### Deactivating the Virtual Environment
When you're done working on the project, you can deactivate the virtual environment:
```bash
deactivate
```
## Contributing
[Add contributing guidelines here]
## License
[Add license information here]
## Project Structure
```
.
├── Dockerfile
├── main
│ ├── api.py
│ ├── app.py
│ ├── config.yaml
│ ├── env_template
│ ├── __init__.py
│ ├── logs
│ │ └── llm_api.log
│ ├── models
│ ├── __pycache__
│ │ ├── api.cpython-39.pyc
│ │ ├── app.cpython-39.pyc
│ │ ├── __init__.cpython-39.pyc
│ │ └── routes.cpython-39.pyc
│ ├── routes.py
│ ├── test_locally.py
│ └── utils
│ ├── errors.py
│ ├── helpers.py
│ ├── __init__.py
│ ├── logging.py
│ ├── __pycache__
│ │ ├── helpers.cpython-39.pyc
│ │ ├── __init__.cpython-39.pyc
│ │ ├── logging.cpython-39.pyc
│ │ └── validation.cpython-39.pyc
│ └── validation.py
├── README.md
└── requirements.txt
```
ERROR:
INFO: 127.0.0.1:60874 - "POST /api/v1/model/download?model_name=microsoft%2FPhi-3.5-mini-instruct HTTP/1.1" 200 OK
2025-01-13 16:18:45,409 - api_routes - INFO - Received request to initialize model: microsoft/Phi-3.5-mini-instruct
2025-01-13 16:18:45,409 - llm_api - INFO - Initializing generation model: microsoft/Phi-3.5-mini-instruct
2025-01-13 16:18:45,412 - llm_api - INFO - Loading model from local path: main/models/Phi-3.5-mini-instruct
The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.
Could not find the bitsandbytes CUDA binary at PosixPath('/home/aurelio/Desktop/Projects/LLMServer/myenv/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_cuda124.so')
g++ (GCC) 14.2.1 20240910
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
2025-01-13 16:18:45,982 - llm_api - ERROR - Failed to initialize generation model microsoft/Phi-3.5-mini-instruct: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
Dynamo is not supported on Python 3.13+
2025-01-13 16:18:45,982 - api_routes - ERROR - Error initializing model: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
Dynamo is not supported on Python 3.13+
INFO: 127.0.0.1:38330 - "POST /api/v1/model/initialize?model_name=microsoft%2FPhi-3.5-mini-instruct HTTP/1.1" 500 Internal Server Error
|