Lee Gao

leegao19's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Upvotes
• Likes

upvoted 20 papers 2 months ago

Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

Paper • 2403.09636 • Published Mar 14 • 2

Video Editing via Factorized Diffusion Distillation

Paper • 2403.09334 • Published Mar 14 • 21

Scattered Mixture-of-Experts Implementation

Paper • 2403.08245 • Published Mar 13 • 1

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Paper • 2401.15947 • Published Jan 29 • 46

Enhancing Multimodal Large Language Models with Vision Detection Models: An Empirical Study

Paper • 2401.17981 • Published Jan 31 • 1

In-Context Learning Creates Task Vectors

Paper • 2310.15916 • Published Oct 24, 2023 • 39

Function Vectors in Large Language Models

Paper • 2310.15213 • Published Oct 23, 2023 • 1

What Algorithms can Transformers Learn? A Study in Length Generalization

Paper • 2310.16028 • Published Oct 24, 2023 • 2

Empower Your Model with Longer and Better Context Comprehension

Paper • 2307.13365 • Published Jul 25, 2023 • 1

Transformer Language Models without Positional Encodings Still Learn Positional Information

Paper • 2203.16634 • Published Mar 30, 2022 • 4

Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level

Paper • 2403.04690 • Published Mar 7 • 1

Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

Paper • 2310.05737 • Published Oct 9, 2023 • 4

An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

Paper • 2403.06764 • Published Mar 11 • 24

MoAI: Mixture of All Intelligence for Large Language and Vision Models

Paper • 2403.07508 • Published Mar 12 • 70

Unfamiliar Finetuning Examples Control How Language Models Hallucinate

Paper • 2403.05612 • Published Mar 8 • 3

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Paper • 2402.13753 • Published Feb 21 • 104

Zoology: Measuring and Improving Recall in Efficient Language Models

Paper • 2312.04927 • Published Dec 8, 2023 • 2

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

Paper • 2311.12351 • Published Nov 21, 2023 • 3

Sequence Parallelism: Long Sequence Training from System Perspective

Paper • 2105.13120 • Published May 26, 2021 • 5

Yi: Open Foundation Models by 01.AI

Paper • 2403.04652 • Published Mar 7 • 59

upvoted a collection 2 months ago

Foundation AI Papers

Collection

Curated List of Must-Reads on LLM reasoning at Temus AI team • 134 items • Updated 23 days ago • 22

upvoted 3 papers 3 months ago

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Paper • 2304.08818 • Published Apr 18, 2023 • 7

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Paper • 2403.03507 • Published Mar 6 • 172

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

Paper • 2309.12288 • Published Sep 21, 2023 • 3

upvoted a collection 3 months ago

Positional embeddings

Collection

13 items • Updated Mar 6 • 2

upvoted 28 papers 3 months ago

AtP*: An efficient and scalable method for localizing LLM behaviour to components

Paper • 2403.00745 • Published Mar 1 • 8

Functional Interpolation for Relative Positions Improves Long Context Transformers

Paper • 2310.04418 • Published Oct 6, 2023 • 4

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Paper • 2006.16236 • Published Jun 29, 2020 • 2

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

Paper • 2402.04347 • Published Feb 6 • 12

Resonance RoPE: Improving Context Length Generalization of Large Language Models

Paper • 2403.00071 • Published Feb 29 • 19

Hydragen: High-Throughput LLM Inference with Shared Prefixes

Paper • 2402.05099 • Published Feb 7 • 16

LongHeads: Multi-Head Attention is Secretly a Long Context Processor

Paper • 2402.10685 • Published Feb 16 • 1

Do Large Language Models Latently Perform Multi-Hop Reasoning?

Paper • 2402.16837 • Published Feb 26 • 24

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

Paper • 2402.16840 • Published Feb 26 • 23

Evaluating Very Long-Term Conversational Memory of LLM Agents

Paper • 2402.17753 • Published Feb 27 • 17

Priority Sampling of Large Language Models for Compilers

Paper • 2402.18734 • Published Feb 28 • 15

Simple linear attention language models balance the recall-throughput tradeoff

Paper • 2402.18668 • Published Feb 28 • 17

Beyond Language Models: Byte Models are Digital World Simulators

Paper • 2402.19155 • Published Feb 29 • 44

Scaling Laws of RoPE-based Extrapolation

Paper • 2310.05209 • Published Oct 8, 2023 • 3

Ring Attention with Blockwise Transformers for Near-Infinite Context

Paper • 2310.01889 • Published Oct 3, 2023 • 8

World Model on Million-Length Video And Language With RingAttention

Paper • 2402.08268 • Published Feb 13 • 33

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Paper • 2309.14509 • Published Sep 25, 2023 • 16

FIT: Far-reaching Interleaved Transformers

Paper • 2305.12689 • Published May 22, 2023 • 1

Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer

Paper • 2310.12442 • Published Oct 19, 2023 • 1

Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention

Paper • 2312.08618 • Published Dec 14, 2023 • 11

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition

Paper • 2402.15220 • Published Feb 23 • 18

A Closer Look at the Limitations of Instruction Tuning

Paper • 2402.05119 • Published Feb 3 • 5

Do Transformers Need Deep Long-Range Memory

Paper • 2007.03356 • Published Jul 7, 2020 • 1

Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models

Paper • 2402.17177 • Published Feb 27 • 87

LoRA+: Efficient Low Rank Adaptation of Large Models

Paper • 2402.12354 • Published Feb 19 • 5

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Paper • 2402.15627 • Published Feb 23 • 31

Mixtures of Experts Unlock Parameter Scaling for Deep RL

Paper • 2402.08609 • Published Feb 13 • 34

In-Context Pretraining: Language Modeling Beyond Document Boundaries

Paper • 2310.10638 • Published Oct 16, 2023 • 26

upvoted 2 collections 3 months ago

RoPE

Collection

3 items • Updated Feb 22 • 2

context

Collection

26 items • Updated 21 days ago • 1

upvoted a paper 3 months ago

Data Engineering for Scaling Language Models to 128K Context

Paper • 2402.10171 • Published Feb 15 • 18

upvoted 3 papers 4 months ago

Efficient Tool Use with Chain-of-Abstraction Reasoning

Paper • 2401.17464 • Published Jan 30 • 15

Rethinking Interpretability in the Era of Large Language Models

Paper • 2402.01761 • Published Jan 30 • 18

Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization

Paper • 2401.07793 • Published Jan 15 • 3

upvoted a collection 4 months ago

Long context

Collection

72 items • Updated 5 days ago • 21