7 15 12

Huiqiang Jiang PRO

iofu728

https://www.microsoft.com/en-us/research/people/hjiang/

AI & ML interests

None yet

Recent Activity

updated a Space about 1 month ago

microsoft/MInference

upvoted a paper about 2 months ago

Differential Transformer

View all activity

Articles

How to Optimize TTFT of 8B LLMs with 1M Tokens to 20s

Jul 21

• 2

Organizations

iofu728's activity

updated a Space about 1 month ago

Running on Zero

🌍

MInference

upvoted a paper about 2 months ago

Differential Transformer

Paper • 2410.05258 • Published Oct 7 • 166

upvoted an article 2 months ago

Article

Fine-tuning LLMs to 1.58bit: extreme quantization made easy

Sep 18

• 203

authored a paper 2 months ago

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Paper • 2409.10516 • Published Sep 16 • 39

upvoted a paper 2 months ago

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Paper • 2409.10516 • Published Sep 16 • 39

commented a paper 2 months ago

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Paper • 2409.10516 • Published Sep 16 • 39 •

upvoted a paper 3 months ago

MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

Paper • 2408.11049 • Published Aug 20 • 12

upvoted an article 3 months ago

Article

A failed experiment: Infini-Attention, and why we should keep trying?

Aug 14

• 50

published an article 4 months ago

Article

How to Optimize TTFT of 8B LLMs with 1M Tokens to 20s

•

Jul 21

• 2

New activity in microsoft/MInference 4 months ago

setup.py error

#1 opened 5 months ago by

NicDev

upvoted 2 articles 5 months ago

Article

RegMix: Data Mixture as Regression for Language Model Pre-training

•

Jul 11

• 10

Article

MInference 1.0: 10x Faster Million Context Inference with a Single GPU

•

Jul 11

• 11

posted an update 5 months ago

Post

1067

Weclome to use MInference, which leverages the dynamic sparse nature of LLMs' attention, which exhibits some static patterns, to speed up the pre-filling for million tokens LLMs. It first determines offline which sparse pattern each head belongs to, then approximates the sparse index online and dynamically computes attention with the optimal custom kernels. This approach achieves up to a 10x speedup for pre-filling on an A100 while maintaining accuracy with 1M tokens.

For more detail please check,
project page: https://aka.ms/MInference
code: https://github.com/microsoft/MInference
paper: MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2407.02490)
hf demo: microsoft/MInference

liked a Space 5 months ago

Running on Zero

🌍

MInference

commented 2 papers 5 months ago

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Paper • 2407.02490 • Published Jul 2 • 23 •

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Paper • 2407.02490 • Published Jul 2 • 23 •

authored a paper 5 months ago

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Paper • 2407.02490 • Published Jul 2 • 23

upvoted a paper 5 months ago

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Paper • 2407.02490 • Published Jul 2 • 23

commented a paper 5 months ago

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Paper • 2407.02490 • Published Jul 2 • 23 •

New activity in zero-gpu-explorers/README 5 months ago

Asking help for using pycuda on Gradio Zero

#78 opened 5 months ago by

iofu728