Instructions to use pinkachu/Estopia-13B-32k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pinkachu/Estopia-13B-32k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pinkachu/Estopia-13B-32k")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pinkachu/Estopia-13B-32k")
model = AutoModelForCausalLM.from_pretrained("pinkachu/Estopia-13B-32k", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pinkachu/Estopia-13B-32k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pinkachu/Estopia-13B-32k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pinkachu/Estopia-13B-32k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pinkachu/Estopia-13B-32k

SGLang

How to use pinkachu/Estopia-13B-32k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pinkachu/Estopia-13B-32k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pinkachu/Estopia-13B-32k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pinkachu/Estopia-13B-32k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pinkachu/Estopia-13B-32k",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use pinkachu/Estopia-13B-32k with Docker Model Runner:
```
docker model run hf.co/pinkachu/Estopia-13B-32k
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

🎸 Estopia-13B-32k

A modernised version of KoboldAI's classic merge for guided narrative writing. Thanks to all the awesome pioneers who made this possible!! ❤️

Source models:

Usage notes

The Estopia model has been tested primarily using the Alpaca format, but with the range of models included likely has some understanding of others.
For settings, the following are recommended for general use:
- Temperature: 0.8-1.2
- Min P: 0.05-0.1
- Max P: 0.92, or 1 if using a Min P greater than 0
- Top K: 0
- Response length: Higher than your usual amount most likely - for example a common value selected is 512.
  - Note: Response lengths are not guaranteed to always be this length. On occasion, responses may be shorter if they convey the response entirely, other times they could be upwards of this value. It depends mostly on the character card, instructions, etc.
- Rep Pen: 1.1
- Rep Pen Range: 2 or 3x your response length
The settings above should provide a generally good experience balancing instruction following and creativity. Generally the higher you set the temperature, the greater the creativity and higher chance of logical errors when providing responses from the AI.
With the differing models inside, this model will not have perfect end of sequence tokens which is a problem many merges can share. While attempts have been made to minimise this, you may occasionally get oddly behaving tokens - this should be possible to resolve with a quick manual edit once and the model should pick up on it.
Chat is one of the least tested areas for this model. It works fairly well, but it can be quite character card dependant.
This is a narrative and prose focused model. As a result, it can and will talk for you if guided to do so (such as asking it to act as a co-author or narrator) within instructions or other contexts. This can be mitigated mostly by adding instructions to limit this, or using chat mode instead.

Downloads last month: 19

Safetensors

Model size

13B params

Tensor type

BF16

Model tree for pinkachu/Estopia-13B-32k

Base model

TheBloke/Llama-2-13B-fp16

Finetuned

KoboldAI/LLaMA2-13B-Estopia

Finetuned

(1)

this model

Quantizations

2 models