Hugging Face's logo

NewTechKids

classroom
Activity Feed Request to join this org

AI & ML interests

None defined yet.

Recent Activity

thomwolf  authored a paper 6 months ago
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
thomwolf  authored a paper 10 months ago
StarCoder 2 and The Stack v2: The Next Generation
thomwolf  authored a paper about 1 year ago
The Stack: 3 TB of permissively licensed source code
View all activity

NewTechKidsAI's activity

thomwolf 
posted an update 17 days ago
view post
Post
4342
We are proud to announce HuggingFaceFW/fineweb-2: A sparkling update to HuggingFaceFW/fineweb with 1000s of 🗣️languages.

We applied the same data-driven approach that led to SOTA English performance in🍷 FineWeb to thousands of languages.

🥂 FineWeb2 has 8TB of compressed text data and outperforms other multilingual datasets in our experiments.

The dataset is released under the permissive 📜 ODC-By 1.0 license, and the 💻 code to reproduce it and our evaluations is public.

We will very soon announce a big community project, and are working on a 📝 blogpost walking you through the entire dataset creation process. Stay tuned!

In the mean time come ask us question on our chat place: HuggingFaceFW/discussion

H/t @guipenedo @hynky @lvwerra as well as @vsabolcec Bettina Messmer @negar-foroutan and @mjaggi
  • 2 replies
·
thomwolf 
posted an update 20 days ago
thomwolf 
posted an update 22 days ago
thomwolf 
posted an update about 1 month ago
thomwolf 
posted an update about 1 month ago
thomwolf 
posted an update 2 months ago
view post
Post
4114
Parents in the 1990: Teach the kids to code
Parents now: Teach the kids to fix the code when it starts walking around 🤖✨
  • 2 replies
·
thomwolf 
authored a paper 6 months ago
thomwolf 
posted an update 7 months ago
view post
Post
4555
[New crazy blog post alert] We are releasing an extensive blog post on the science of creating high quality web-scale datasets, detailing all the steps and learnings that came in our recent 15 trillion tokens 🍷FineWeb release

Inspired by the distill.pub interactive graphics papers, we settled to write the most extensive, enjoyable and in-depth tech report we could draft on so prepare for a 45-mmin read with interactive graphics and all.

And it's not all, in this article we also introduce 📚FineWeb-Edu a filtered subset of Common Crawl with 1.3T tokens containing only web pages with very high educational content. Up to our knowledge, FineWeb-Edu out-performs all openly release web-scale datasets by a significant margin on knowledge- and reasoning-intensive benchmarks like MMLU, ARC, and OpenBookQA

We also make a number of surprising observations on the "quality" of the internet it-self which may challenge some of the general assumptions on web data (not saying more, I'll let you draw your conclusions ;)

HuggingFaceFW/blogpost-fineweb-v1
  • 1 reply
·
thomwolf 
posted an update 8 months ago
view post
Post
4868
Is is time for the open-source AI robots revolution 🚀?

With @haixuantao and @Leyo we’ve been playing with a low-cost DJI robot controlled by three local open-source AI models (Whisper, Idefics2, Parler-TTS - all Apache2) and orchestrated by Dora-cs.

Links to find all the hardware/software we used in the demo:
- robot control framework – dora-rs: https://github.com/dora-rs/dora
- speech-to-text model – whisper: openai/whisper-base
- vision-text model – Idefics2: HuggingFaceM4/idefics2-8b-AWQ
- text-to-speech model – ParlerTTS mini: parler-tts/parler_tts_mini_v0.1
- robot: https://dji.com/robomaster-s1
- code gist: https://gist.github.com/haixuanTao/860e1740245dc2c8dd85b496150a9320
- Larger codebase: dora-rs/dora-idefics2
- laptop/pc: any with a recent GPU card (our has a RTX 4090)

Enjoy!
·
thomwolf 
posted an update 9 months ago
view post
Post
3123
Very interesting model just released by MyShell: jetmoe/jetmoe-8b . It's a 8B-parameters MoE LLM so 2.2B active parameters, really efficient.

Main characteristics:
- impressive performances for its size (beating meta-llama/Llama-2-7b and huggyllama/llama-13b)
- combine Mixture of Attention heads (MoA) and Mixture of MLP Experts (MoE) – 8 experts with 2 being active for each token
- trained on a rather limited 1.25T tokens from publicly available datasets – training recipe follows the MiniCPM's two-phases training method => first time I see this for a 2B+ model
- $100k to train
- open weights - open sharing of recipes - open dataset - open code => ♡
- still interesting room to improve performances (be it only by training longer)

Links:
- report: https://research.myshell.ai/jetmoe
- model: jetmoe/jetmoe-8b
- code: https://github.com/myshell-ai/JetMoE

Note: I actually detailed all of the MiniCPM schedule, Mixture-of-expert (MoE) and many of the datasets used in this work in my recent little guide to building LLMs in 2024, so feel free to check it out if you want to learn more on these topics: https://www.youtube.com/watch?v=2-SPH9hIKT8
  • 1 reply
·
thomwolf 
posted an update 9 months ago
view post
Post
1993
Little know gem: the Open-source Cookbook

A collection of notebooks for building practical AI applications using open-source tools and models: https://lnkd.in/e6m6Jmwu

Doc: https://lnkd.in/e3FE6TUq

Currently contains 16 notebooks in English (and some in Chinese):
1. Using LLM-as-a-judge 🧑‍⚖️ for an automated and versatile evaluation
2. Create a legal preference dataset
3. Suggestions for Data Annotation with SetFit in Zero-shot Text Classification
4. Implementing semantic cache to improve a RAG system
5. Building A RAG Ebook “Librarian” Using LlamaIndex
6. Stable Diffusion Interpolation
7. Building A RAG System with Gemma, MongoDB and Open Source Models
8. Prompt Tuning with PEFT Library
9. Migrating from OpenAI to Open LLMs Using TGI’s Messages API
10. Automatic Embeddings with TEI through Inference Endpoints
11. Simple RAG for GitHub issues using Hugging Face Zephyr and LangChain
12. Embedding multimodal data for similarity search using 🤗 transformers, 🤗 datasets and FAISS
13. Fine-tuning a Code LLM on Custom Code on a single GPU
14. RAG Evaluation Using Synthetic data and LLM-As-A-Judge
15. Advanced RAG on HuggingFace documentation using LangChain
16. Detecting Issues in a Text Dataset with Cleanlab
thomwolf 
posted an update 9 months ago
view post
Post
5043
A Little guide to building Large Language Models in 2024

This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.

In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
* finding, preparing and evaluating web scale data
* understanding model parallelism and efficient training
* fine-tuning/aligning models
* fast inference

There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.

Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
*datatrove for all things web-scale data preparation: https://github.com/huggingface/datatrove
*nanotron for lightweight 4D parallelism LLM training: https://github.com/huggingface/nanotron
*lighteval for in-training fast parallel LLM evaluations: https://github.com/huggingface/lighteval

Here is the link to watch the lecture on Youtube: https://www.youtube.com/watch?v=2-SPH9hIKT8
And here is the link to the Google slides: https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p

Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
  • 2 replies
·
thomwolf 
posted an update 9 months ago
thomwolf 
authored a paper 10 months ago
thomwolf 
authored 6 papers about 1 year ago