Instructions to use epfml/landmark-attention-llama7b-wdiff with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use epfml/landmark-attention-llama7b-wdiff with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="epfml/landmark-attention-llama7b-wdiff")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("epfml/landmark-attention-llama7b-wdiff")
model = AutoModelForCausalLM.from_pretrained("epfml/landmark-attention-llama7b-wdiff", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use epfml/landmark-attention-llama7b-wdiff with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "epfml/landmark-attention-llama7b-wdiff"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "epfml/landmark-attention-llama7b-wdiff",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/epfml/landmark-attention-llama7b-wdiff

SGLang

How to use epfml/landmark-attention-llama7b-wdiff with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "epfml/landmark-attention-llama7b-wdiff" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "epfml/landmark-attention-llama7b-wdiff",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "epfml/landmark-attention-llama7b-wdiff" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "epfml/landmark-attention-llama7b-wdiff",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use epfml/landmark-attention-llama7b-wdiff with Docker Model Runner:
```
docker model run hf.co/epfml/landmark-attention-llama7b-wdiff
```

LLaMA-7B + Landmark Attention

This repo hosts the weight diff between LLaMA 7B trained with landmark attention for 15000 steps on RedPajama and the original model. Please visit the Github repository for further instructions on how to recover the full weights and how to use them.

Github repository: https://github.com/epfml/landmark-attention

Downloads last month: 55

epfml
/

landmark-attention-llama7b-wdiff

LLaMA-7B + Landmark Attention

Spaces using epfml/landmark-attention-llama7b-wdiff 3