Furu Wei

thegenerality's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Upvotes
• Likes

upvoted a paper 18 days ago

BitNet a4.8: 4-bit Activations for 1-bit LLMs

Paper • 2411.04965 • Published 18 days ago • 63

authored 3 papers about 2 months ago

Data Selection via Optimal Control for Language Models

Paper • 2410.07064 • Published Oct 9 • 8

Self-Boosting Large Language Models with Synthetic Preference Data

Paper • 2410.06961 • Published Oct 9 • 15

Differential Transformer

Paper • 2410.05258 • Published Oct 7 • 166

upvoted a paper about 2 months ago

Differential Transformer

Paper • 2410.05258 • Published Oct 7 • 166

upvoted a paper 4 months ago

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

Paper • 2407.10969 • Published Jul 15 • 20

authored a paper 5 months ago

Autoregressive Speech Synthesis without Vector Quantization

Paper • 2407.08551 • Published Jul 11 • 14

upvoted a paper 5 months ago

Direct Preference Knowledge Distillation for Large Language Models

Paper • 2406.19774 • Published Jun 28 • 21

authored 2 papers 5 months ago

Direct Preference Knowledge Distillation for Large Language Models

Paper • 2406.19774 • Published Jun 28 • 21

Instruction Pre-Training: Language Models are Supervised Multitask Learners

Paper • 2406.14491 • Published Jun 20 • 85

authored 2 papers 6 months ago

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

Paper • 2406.05370 • Published Jun 8 • 14

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Paper • 2405.12130 • Published May 20 • 46

liked a model 7 months ago

microsoft/kosmos-2.5

Text2Text Generation • Updated Aug 28 • 2.5k • 171

authored a paper 7 months ago

Multi-Head Mixture-of-Experts

Paper • 2404.15045 • Published Apr 23 • 59

liked a model 8 months ago

1bitLLM/bitnet_b1_58-3B

Text Generation • Updated Mar 29 • 3.02k • 240

Reacted to shumingma's post with 🚀 8 months ago

view post

Post

2605

Reply

The Era of 1-bit LLMs: Training Tips, Code and FAQ

https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf

We present details and tips for training 1-bit LLMs. We also provide additional experiments and results that were not reported and responses to questions regarding the "The-Era-of-1-bit-LLM" paper. Finally, we include the official PyTorch implementation of BitNet (b1.58 and b1) for future research and development of 1-bit LLMs.

• 
• 2 replies

·

❤️

14

14

🚀

7

7

🔥

4

4

+

authored 4 papers 9 months ago

MathScale: Scaling Instruction Tuning for Mathematical Reasoning

Paper • 2403.02884 • Published Mar 5 • 15

Towards Optimal Learning of Language Models

Paper • 2402.17759 • Published Feb 27 • 16

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Paper • 2402.17764 • Published Feb 27 • 603

Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models

Paper • 2402.13064 • Published Feb 20 • 47