Hugging Face's logo
Jędrzej Grabala's picture
19 87

Jędrzej Grabala

jgitsolutions
ksiabani's profile picture 21world's profile picture Mi6paulino's profile picture
·
https://jgitsol.github.io

AI & ML interests

Local Drive Human Overseered System of Agents, LLMs, Langchains & other useful stuff on mid-to-low-end of commercial hardware.

Recent Activity

Reacted to anakin87's post with 👀 7 days ago
Ok, you're finally convinced that synthetic data works... ⚗️ 𝐍𝐨𝐰 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐞 𝐚𝐧 𝐢𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐢𝐧 𝐚 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐨𝐭𝐡𝐞𝐫 𝐭𝐡𝐚𝐧 𝐄𝐧𝐠𝐥𝐢𝐬𝐡. But how do you get started? I explore how to do this with Magpie in my new article https://huggingface.co/blog/anakin87/multilingual-magpie --- 🐦‍⬛ 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐌𝐚𝐠𝐩𝐢𝐞? It's a recent technique for creating synthetic instruction datasets. Magpie is based on a simple but ingenious idea 👇 if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction Here's an example: model: Llama-3-8B-Instruct pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>" generated user instruction: "What are some of the responsibilities of a commercial pilot?" You can then feed this instruction back into the same model to get the assistant response. By repeating this process, it's possible to generate large synthetic datasets with relatively little effort. 🪄 The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model. 🧗𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐧𝐨𝐧-𝐄𝐧𝐠𝐥𝐢𝐬𝐡 𝐝𝐚𝐭𝐚 Most Language Models are primarily trained on English texts, so they tend to produce data in English. How can we overcome this? Earlier approaches were complex or costly. Then @mrm8488 found a simple solution: add the target language to the pre-query template. For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:". This method works for Spanish and German! ❌ Unfortunately, it does not work well for other languages (🇮🇹, 🇳🇱, ...) 👇
Reacted to yongchanghao's post with 👀 7 days ago
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below). Read more about the work at https://huggingface.co/papers/2410.20650
Reacted to yongchanghao's post with 🔥 7 days ago
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below). Read more about the work at https://huggingface.co/papers/2410.20650
View all activity

Organizations

LangChain Agents Hub's profile picture LangChainDatasets's profile picture ZeroGPU Explorers's profile picture Dev Mode Explorers's profile picture

jgitsolutions's activity

liked a Space 11 days ago
liked a Space 12 days ago
liked 3 Spaces 13 days ago
liked 2 models 20 days ago
liked 4 Spaces about 1 month ago
liked a dataset about 1 month ago
liked 4 Spaces about 2 months ago
liked a model about 2 months ago
liked a Space about 2 months ago
liked 2 Spaces 2 months ago