Travis King's picture

Travis King

travisking

·

AI & ML interests

have you heard of generative AI?

Recent Activity

upvoted a paper 1 day ago

Phi-4 Technical Report

Reacted to singhsidhukuldeep's post with 👀 2 days ago

Exciting new research alert! 🚀 A groundbreaking paper titled "Understanding LLM Embeddings for Regression" has just been released, and it's a game-changer for anyone working with large language models (LLMs) and regression tasks. Key findings: 1. LLM embeddings outperform traditional feature engineering in high-dimensional regression tasks. 2. LLM embeddings preserve Lipschitz continuity over feature space, enabling better regression performance. 3. Surprisingly, factors like model size and language understanding don't always improve regression outcomes. Technical details: The researchers used both T5 and Gemini model families to benchmark embedding-based regression. They employed a key-value JSON format for string representations and used average-pooling to aggregate Transformer outputs. The study introduced a novel metric called Normalized Lipschitz Factor Distribution (NLFD) to analyze embedding continuity. This metric showed a high inverse relationship between the skewedness of the NLFD and regression performance. Interestingly, the paper reveals that applying forward passes of pre-trained models doesn't always significantly improve regression performance for certain tasks. In some cases, using only vocabulary embeddings without a forward pass yielded comparable results. The research also demonstrated that LLM embeddings are dimensionally robust, maintaining strong performance even with high-dimensional data where traditional representations falter. This work opens up exciting possibilities for using LLM embeddings in various regression tasks, particularly those with high degrees of freedom. It's a must-read for anyone working on machine learning, natural language processing, or data science!

liked a dataset 2 days ago

fozziethebeat/alpaca_messages_2k_dpo_test

View all activity

Organizations

None yet

travisking's activity

upvoted a paper 1 day ago

Phi-4 Technical Report

Paper • 2412.08905 • Published 3 days ago • 50

upvoted a paper 2 days ago

Evaluating Language Models as Synthetic Data Generators

Paper • 2412.03679 • Published 10 days ago • 39

upvoted a paper 3 days ago

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

Paper • 2411.10640 • Published 29 days ago • 44

upvoted a collection about 1 month ago

SmolLM2

State-of-the-art compact LLMs for on-device applications: 1.7B, 360M, 135M • 15 items • Updated 12 days ago • 191

upvoted an article 3 months ago

Article

SetFit: Efficient Few-Shot Learning Without Prompts

Sep 26, 2022

• 20

upvoted 2 collections 3 months ago

Zeroshot Classifiers

These are my current best zeroshot classifiers. Some of my older models are downloaded more often, but the models in this collection are newer/better. • 11 items • Updated Apr 3 • 113

OLMoE

Artifacts for open mixture-of-experts language models. • 13 items • Updated 17 days ago • 27

upvoted 2 articles 4 months ago

Article

A failed experiment: Infini-Attention, and why we should keep trying?

Aug 14

• 51

Article

Chat Templates: An End to the Silent Performance Killer

Oct 3, 2023

• 14

upvoted a paper 4 months ago

Gemma 2: Improving Open Language Models at a Practical Size

Paper • 2408.00118 • Published Jul 31 • 75

upvoted a collection 5 months ago

Bad Data Toolbox

PleIAs collection of models for the data processing of challenging document and data sources. • 5 items • Updated Jul 18 • 15

upvoted an article 5 months ago

Article

Advanced RAG: Fine-Tune Embeddings from HuggingFace for RAG

By

•

Jul 5

• 4

upvoted a collection 5 months ago

MatMulfree LM

Pre-trined models for Matmulfree LM. • 4 items • Updated Jun 10 • 25

upvoted a paper 6 months ago

GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks

Paper • 2406.12925 • Published Jun 14 • 23

upvoted 2 collections 7 months ago

A little guide to building Large Language Models in 2024

Resources mentioned by @thomwolf in https://x.com/Thom_Wolf/status/1773340316835131757 • 19 items • Updated Apr 1 • 14

OCR Quality Classifiers

predict OCR quality • 4 items • Updated May 12 • 2

upvoted an article 8 months ago

Article

GaLore: Advancing Large Model Training on Consumer-grade Hardware

Mar 20

• 26

upvoted a collection 8 months ago

Performance LLMs - Base Models

18 items • Updated 4 days ago • 8