OTel-2.0-LLM-31B-IT-FP8

Model Description

This is an FP8 dynamic quantized version of farbodtavakkoli/OTel-2.0-LLM-31B-IT. The model was quantized to FP8 using Neural Magic's llmcompressor to optimize memory footprint and maximize inference speed on compatible GPUs (such as NVIDIA H100, A100, and RTX 40-series) using vLLM.

• Developed by: Ukkathva (Quantized version), original by Farbod Tavakkoli
• Model type: Causal Language Model
• Language(s): English
• Finetuned from model: farbodtavakkoli/OTel-2.0-LLM-31B-IT

How to Get Started with the Model

This model is best used with vLLM for highly optimized FP8 inference.

pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(model="ukkathva/OTel-2.0-LLM-31B-IT-FP8")
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)

prompts = ["Explain the architecture of an OTel (OpenTelemetry) pipeline."]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")

Quantization Details

The model was quantized using llmcompressor with the FP8_DYNAMIC scheme applied to all Linear layers, while intentionally skipping the lm_head layer to maintain maximum accuracy.