semvec/pypi-packages huggingface dataset:
cosmian-py-ggml 1.2.0
Python bindings for GGML
License: MIT
Requires Python: >=3.8,<4.0

# Cosmian Python bindings for [`ggml`](https://github.com/ggerganov/ggml)

[![PyPI version](https://badge.fury.io/py/cosmian_py_ggml.svg)](https://badge.fury.io/py/cosmian_py_ggml)

This library provides python bindings for the [examples models](https://github.com/ggerganov/ggml/tree/master/examples) of ggml.

Using this library, you can deploy state-of-the-art LLMs on Intel SGX hardware with [Cosmian MSE](https://cosmian.com/cosmian-microservice-encryption/).

## Available models architecture

- [gpt-neox](https://huggingface.co/docs/transformers/model_doc/gpt_neox)

```python
from cosmian_py_ggml import GPTNeoX
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
model = GPTNeoX("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))
```

- [falcon](https://huggingface.co/blog/falcon)

```python
from cosmian_py_ggml import Falcon
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b")
model = Falcon("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))
```

- [mpt](https://github.com/mosaicml/llm-foundry#mpt)

```python
from cosmian_py_ggml import MPT
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mosaicml/mpt-30b")
model = MPT("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))
```

## Build from sources

```bash
git clone --recursive git@github.com:Cosmian/cosmian_py_ggml.git
scripts/build.sh
```