---

base_model: allenai/AstaBrief_8B library_name: transformers license: apache-2.0 language: - en pipeline_tag: text-generation tags: - fp8 - dynamic-quantization - llm-compressor - vllm - qwen3 - deep-research - science - scholar-qa quantized_by: jjjssjs

AstaBrief_8B-FP8-Dynamic

This repository contains the FP8 Dynamic Quantized version of allenai/AstaBrief_8B (Qwen3-8B architecture).

By quantizing weights and activations to FP8 with dynamic per-token scaling, this model offers near-lossless accuracy compared to the original BF16 model while significantly reducing memory footprint and maximizing decoding throughput on modern GPU architectures (NVIDIA Ada Lovelace, Hopper, Blackwell, and newer).

---

📌 Key Highlights

• Base Model: allenai/AstaBrief_8B
• Quantization Type: FP8 (Dynamic Activation / Static or Dynamic Weight Scaling)
• Primary Use Case: Fast, single-pass scientific paper summarization and report generation with inline citations.
• Performance: Near 100% preservation of original BF16 citation recall and precision metrics with up to 1.8x–2.5x speedup on FP8 Tensor Cores.
• Context Length: Supports up to 16,000 tokens context window.

---

💻 Quickstart Guide

1. High-Throughput Serving via vLLM (Recommended)

vLLM natively accelerates FP8 execution with FP8 Tensor Cores.

from vllm import LLM, SamplingParams

model_id = "jjjssjs/AstaBrief_8B-FP8-Dynamic"

llm = LLM(
    model=model_id,
    tensor_parallel_size=1,
    max_model_len=16000
)

# Recommended input prompt structure for AstaBrief
prompt = """Write a detailed research report with inline citations based on the provided query and retrieved literature excerpts.

Query: What are the primary failure modes of transformer-based LLMs in complex mathematical reasoning?

References:
[1] Excerpt from literature source A...
[2] Excerpt from literature source B...
"""

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=2048
)

outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)