Text Generation
Transformers
Safetensors
Chinese
English
joyai_llm_flash
conversational
custom_code
fp8
Instructions to use jdopensource/JoyAI-LLM-Flash-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jdopensource/JoyAI-LLM-Flash-FP8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="jdopensource/JoyAI-LLM-Flash-FP8", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("jdopensource/JoyAI-LLM-Flash-FP8", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use jdopensource/JoyAI-LLM-Flash-FP8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jdopensource/JoyAI-LLM-Flash-FP8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jdopensource/JoyAI-LLM-Flash-FP8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/jdopensource/JoyAI-LLM-Flash-FP8
- SGLang
How to use jdopensource/JoyAI-LLM-Flash-FP8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "jdopensource/JoyAI-LLM-Flash-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jdopensource/JoyAI-LLM-Flash-FP8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "jdopensource/JoyAI-LLM-Flash-FP8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jdopensource/JoyAI-LLM-Flash-FP8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use jdopensource/JoyAI-LLM-Flash-FP8 with Docker Model Runner:
docker model run hf.co/jdopensource/JoyAI-LLM-Flash-FP8
| from __future__ import annotations | |
| import select | |
| import socket | |
| import sys | |
| def is_socket_readable(sock: socket.socket | None) -> bool: | |
| """ | |
| Return whether a socket, as identifed by its file descriptor, is readable. | |
| "A socket is readable" means that the read buffer isn't empty, i.e. that calling | |
| .recv() on it would immediately return some data. | |
| """ | |
| # NOTE: we want check for readability without actually attempting to read, because | |
| # we don't want to block forever if it's not readable. | |
| # In the case that the socket no longer exists, or cannot return a file | |
| # descriptor, we treat it as being readable, as if it the next read operation | |
| # on it is ready to return the terminating `b""`. | |
| sock_fd = None if sock is None else sock.fileno() | |
| if sock_fd is None or sock_fd < 0: # pragma: nocover | |
| return True | |
| # The implementation below was stolen from: | |
| # https://github.com/python-trio/trio/blob/20ee2b1b7376db637435d80e266212a35837ddcc/trio/_socket.py#L471-L478 | |
| # See also: https://github.com/encode/httpcore/pull/193#issuecomment-703129316 | |
| # Use select.select on Windows, and when poll is unavailable and select.poll | |
| # everywhere else. (E.g. When eventlet is in use. See #327) | |
| if ( | |
| sys.platform == "win32" or getattr(select, "poll", None) is None | |
| ): # pragma: nocover | |
| rready, _, _ = select.select([sock_fd], [], [], 0) | |
| return bool(rready) | |
| p = select.poll() | |
| p.register(sock_fd, select.POLLIN) | |
| return bool(p.poll(0)) | |