Open-source embeddings and LLMs outperform Gemini and OpenAI for Web Navigation while being faster and cheaper
โข
7
Daniel Huynh PRO
dhuynh95
AI & ML interests
None yet
Recent Activity
liked
a model
9 days ago
microsoft/OmniParser
updated
a collection
13 days ago
cool-papers
upvoted
a
paper
13 days ago
Articles
Organizations
dhuynh95's activity
posted
an
update
6 months ago
Post
1596
๐ชBuild an information retrieval Agent that can beat Gemini and OpenAI using open-source Large Action Model framework!
In this video, we ask to different proprietary Conversational AI the question:
โWhat is the most trendy recent paper on Llava models on Hugging Face papers? Provide the date and a summary of the paperโ, and the results are interesting!
โGemini: found a paper from Jan 29, 2024
โOpenAI: found a paper from October 2023
โYou.com: found a paper from Jan 29 2024
โ LaVague: found the latest paper (ConvLlaVA which is dope by the way https://arxiv.org/abs/2405.15738)!
The best? Our solution fits a few ines of code with our open-source framework! I will share how we built that agent during our webinar on AI Web Agents, this Thursday 30th May at 9 am PST (https://lu.ma/m8fzmb3q) so donโt miss it ๐
You can also start playing with our framework: https://github.com/lavague-ai/LaVague
In this video, we ask to different proprietary Conversational AI the question:
โWhat is the most trendy recent paper on Llava models on Hugging Face papers? Provide the date and a summary of the paperโ, and the results are interesting!
โGemini: found a paper from Jan 29, 2024
โOpenAI: found a paper from October 2023
โYou.com: found a paper from Jan 29 2024
โ LaVague: found the latest paper (ConvLlaVA which is dope by the way https://arxiv.org/abs/2405.15738)!
The best? Our solution fits a few ines of code with our open-source framework! I will share how we built that agent during our webinar on AI Web Agents, this Thursday 30th May at 9 am PST (https://lu.ma/m8fzmb3q) so donโt miss it ๐
You can also start playing with our framework: https://github.com/lavague-ai/LaVague
reacted to
m-ric's
post with ๐
8 months ago
Post
1798
๐๐ก๐ ๐ซ๐๐ญ๐ฎ๐ซ๐ง ๐จ๐ ๐ญ๐ก๐ ๐๐๐๐ฌ โ ๐๐๐ฐ ๐๐๐ฆ๐๐-๐๐๐ฌ๐๐ ๐๐ซ๐๐ก๐ข๐ญ๐๐๐ญ๐ฎ๐ซ๐ "๐๐๐ฆ๐๐"
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
posted
an
update
8 months ago
Post
1757
๐LaVague can compile Action Plans into actionable code to browse the internet!
In this example, you can see how an action plan with natural language instructions can be โcompiledโ into executable Selenium code!
๐คThis shows the potential of #LAM (Large Action Models) to perform actions for us and automate mechanical tasks.
This example leverages a local embedding model and OpenAI GPT-3.5, but we support many options, including local ones with Gemma!
You can try this in our docs: https://docs.lavague.ai/en/latest/
LaVague is an open-source Large Action Model framework to automate automation. If you are interested in helping us on our mission to democratize automation tooling for devs, donโt hesitate to visit our GitHub (https://github.com/lavague-ai/LaVague) or Discord (https://discord.gg/SDxn9KpqX9)!
In this example, you can see how an action plan with natural language instructions can be โcompiledโ into executable Selenium code!
๐คThis shows the potential of #LAM (Large Action Models) to perform actions for us and automate mechanical tasks.
This example leverages a local embedding model and OpenAI GPT-3.5, but we support many options, including local ones with Gemma!
You can try this in our docs: https://docs.lavague.ai/en/latest/
LaVague is an open-source Large Action Model framework to automate automation. If you are interested in helping us on our mission to democratize automation tooling for devs, donโt hesitate to visit our GitHub (https://github.com/lavague-ai/LaVague) or Discord (https://discord.gg/SDxn9KpqX9)!
posted
an
update
8 months ago
Post
Hello World! This post is written by the Large Action Model framework LaVague! Find out more on https://github.com/mithril-security/LaVague
Edit: Here is the video of ๐LaVague posting this. This is quite meta
Edit: Here is the video of ๐LaVague posting this. This is quite meta
- 2 replies
replied to
their
post
9 months ago
Thanks! Preparing a tutorial too to explain how we managed to have a working solution in <150 lines of code :D
posted
an
update
9 months ago
Post
๐ Released #LaVague, fullly open-source AI pipeline to turn natural language into browser actions!
In less than 150 lines of code (RAG with local embedding + Zephyr-7b-Gemma locally or Mixtral on HF Inference API), it generates #Selenium code from user query. In this GIF you can see it follow user instructions to command a browser to browse HF website!
Try it on Colab: colab.research.google.com/github/dhuynh95/LaVague/blob/main/LaVague.ipynb
GitHub: github.com/dhuynh95/LaVague
Pretty exciting how it becomes possible to create an AI assistant that could perform actions for us, such as logging on gov accounts, fill forms, or pull personal information!
It was quite fun to hack in the weekend using open-source tools, from @huggingface local embedding with transformers for local inference or HF Inference API, to RAG with @llama_index, through @MistralAI Mixtral model!
Some challenges: to make it run on Colab for the #GPU Poors, I first resorted to @huggingface Inference API with Mixtral as it was the only model good enough (gemma-7b did not make it and refused to produce code). But after some experimentations, I managed to make it work a local Zephyr-7b-Gemma so that people could run this assistant fully locally!
Because I used an off-the-shelf model, I had to improve performance with few-shot learning and Chain Of Thought, which managed to generate appropriate code!
I hope this project will herald a new dawn where transparent, private and local AI assistants help automate menial but critical tasks, such as helping fill taxes, book accomodation, or research information for us.
In less than 150 lines of code (RAG with local embedding + Zephyr-7b-Gemma locally or Mixtral on HF Inference API), it generates #Selenium code from user query. In this GIF you can see it follow user instructions to command a browser to browse HF website!
Try it on Colab: colab.research.google.com/github/dhuynh95/LaVague/blob/main/LaVague.ipynb
GitHub: github.com/dhuynh95/LaVague
Pretty exciting how it becomes possible to create an AI assistant that could perform actions for us, such as logging on gov accounts, fill forms, or pull personal information!
It was quite fun to hack in the weekend using open-source tools, from @huggingface local embedding with transformers for local inference or HF Inference API, to RAG with @llama_index, through @MistralAI Mixtral model!
Some challenges: to make it run on Colab for the #GPU Poors, I first resorted to @huggingface Inference API with Mixtral as it was the only model good enough (gemma-7b did not make it and refused to produce code). But after some experimentations, I managed to make it work a local Zephyr-7b-Gemma so that people could run this assistant fully locally!
Because I used an off-the-shelf model, I had to improve performance with few-shot learning and Chain Of Thought, which managed to generate appropriate code!
I hope this project will herald a new dawn where transparent, private and local AI assistants help automate menial but critical tasks, such as helping fill taxes, book accomodation, or research information for us.
-
-
- 5 replies
posted
an
update
10 months ago
Post
โจ In-context learning is all you need!
This super interesting paper shows that fine-tuning with #SFT or #RLHF only helps on the form but does not impact knowledge or reasoning abilities, and in some cases, actually decreases performance!
They tested it with Mistral-base vs Mistral FT-ed, as well as Llama 2 70b base and FT-ed and results are consistent.
Providing the right prompt to the base model actually makes the model better and has 0 training cost!
Paper: https://arxiv.org/abs/2312.01552
This super interesting paper shows that fine-tuning with #SFT or #RLHF only helps on the form but does not impact knowledge or reasoning abilities, and in some cases, actually decreases performance!
They tested it with Mistral-base vs Mistral FT-ed, as well as Llama 2 70b base and FT-ed and results are consistent.
Providing the right prompt to the base model actually makes the model better and has 0 training cost!
Paper: https://arxiv.org/abs/2312.01552
reacted to
abidlabs's
post with โค๏ธ
10 months ago
Post
Just out: new custom Gradio component specifically designed for code completion models ๐ฅ
- 1 reply
Pretty cool stuff! Maybe you should do a leaderboard of major datasets and their leakage score
reacted to
lbourdois's
post with ๐คฏ
10 months ago
Post
Let me introduce you LLE: Leaks, leaks everywhere!
A quick experiment I've carried out on around 600 datasets from the HF Hub, the results are stored in lbourdois/LLE, and the methodology is described in
https://huggingface.co/blog/lbourdois/lle
A quick experiment I've carried out on around 600 datasets from the HF Hub, the results are stored in lbourdois/LLE, and the methodology is described in
https://huggingface.co/blog/lbourdois/lle
- 7 replies
reacted to
Locutusque's
post with โค๏ธ
10 months ago
Post
Introducing the "UltraTextbooks" dataset ๐๐
Check it out here: Locutusque/UltraTextbooks
๐ A comprehensive collection of high-quality synthetic and human-written textbooks
๐จโ๐ Spanning various subjects and programming languages
๐ง Designed for advanced NLP tasks like language modeling, educational QA, text summarization, and content generation for edu purposes
๐ Future expansions planned with additional data sources to enhance the corpus
๐ Data composition highlights ๐
- Blend of synthetic and human-written material
- Includes topics from general edu to specialized areas
- Structured with field "text"
๐งฉ Data collection from various Hugging Face datasets, guided by a diverse and comprehensive curation rationale
๐ง Limitations may exist, so report any issues you encounter
Check it out here: Locutusque/UltraTextbooks
๐ A comprehensive collection of high-quality synthetic and human-written textbooks
๐จโ๐ Spanning various subjects and programming languages
๐ง Designed for advanced NLP tasks like language modeling, educational QA, text summarization, and content generation for edu purposes
๐ Future expansions planned with additional data sources to enhance the corpus
๐ Data composition highlights ๐
- Blend of synthetic and human-written material
- Includes topics from general edu to specialized areas
- Structured with field "text"
๐งฉ Data collection from various Hugging Face datasets, guided by a diverse and comprehensive curation rationale
๐ง Limitations may exist, so report any issues you encounter
- 2 replies
reacted to
gsarti's
post with ๐
10 months ago
Post
๐ Today's pick in Interpretability & Analysis of LMs: Black-Box Access is Insufficient for Rigorous AI Audits by
@stecas
@carsonezell
et al.
Audits conducted on AI systems can identify potential risks and ensure their compliance to safety requirements. Authors categorise audits based on the access to model-related resources (black, grey, white and out-of-the box) and highlight how levels of transparency on audited AI system enable broader and more effective auditing procedures. Technical, physical, and legal safeguards for performing audits are also introduced to ensure minimal security risks for audited companies. Authors conclude that transparency on the type of auditorsโ access and methods is a pre-requisite to correctly interpret audit results, and white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.
๐ Paper: Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446)
๐ Further readings:
๐Taxonomy of AI system access: https://bit.ly/struct-access
๐ปAn API for transparent science on Black-box AI (NNsight): https://nnsight.net/about
Audits conducted on AI systems can identify potential risks and ensure their compliance to safety requirements. Authors categorise audits based on the access to model-related resources (black, grey, white and out-of-the box) and highlight how levels of transparency on audited AI system enable broader and more effective auditing procedures. Technical, physical, and legal safeguards for performing audits are also introduced to ensure minimal security risks for audited companies. Authors conclude that transparency on the type of auditorsโ access and methods is a pre-requisite to correctly interpret audit results, and white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.
๐ Paper: Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446)
๐ Further readings:
๐Taxonomy of AI system access: https://bit.ly/struct-access
๐ปAn API for transparent science on Black-box AI (NNsight): https://nnsight.net/about
posted
an
update
10 months ago
Post
Fascinating paper by Rand shows that there is no statistically significant difference between using LLMs or regular internet to craft operational plans for bioweapons!
This is the first paper that actually studies the impact of AI on bioweapons from an operational perspective and looks at the big question: is AI any better than just using public data on the Internet?
As most of the data is most likely out there, an LLM would just be a more efficient tool to come up with the relevant information, but it seems that its impact is limited.
https://www.rand.org/pubs/research_reports/RRA2977-2.html
This is the first paper that actually studies the impact of AI on bioweapons from an operational perspective and looks at the big question: is AI any better than just using public data on the Internet?
As most of the data is most likely out there, an LLM would just be a more efficient tool to come up with the relevant information, but it seems that its impact is limited.
https://www.rand.org/pubs/research_reports/RRA2977-2.html
- 1 reply
replied to
santiviquez's
post
10 months ago
Love this paper too! It's simple yet powerful and applicable to black box models.
I actually have a space to demonstrate it: https://huggingface.co/spaces/mithril-security/hallucination_detector
I also dig into it on an HF Blog post: https://huggingface.co/blog/dhuynh95/automatic-hallucination-detection
posted
an
update
10 months ago
Post
โ
New paper to ensure valid LLM output with SOTA LLMs like GPT4 by mixing it with OSS LLMs
Paper: arxiv.org/abs/2401.09967
Great paper showing how strong proprietary AI like #GPT4 can be paired with #OSS LLM to ensure LLM output validity, e.g. valid JSON.
Many devs complain that #LLMs cannot be reliably used in production if the output is not valid, for instance, if one wants to use LLMs to generate SQL queries or JSON, it is crucial that the output is valid.
Frameworks have arisen to constrain the outputs of the LLM to follow some constraints, like outlines (https://github.com/outlines-dev/outlines), but they assume access to logits.
This makes them incompatible with proprietary LLMs like GPT4 that donโt share logits, so one can only use open-source LLMs that are much less performant.
This paper shows how can use powerful proprietary LLMs like GPT4 to create a first unconstrained sketch and refine it using an OSS model like Llama 2 where logits are accessible, to rewrite the sketch following some specific constraints.
They show that GPT4 Precision can be increased by 14% (43% before, 57% after), by boosting it with constrained output on information extraction on Wiki-NRE!
Paper: arxiv.org/abs/2401.09967
Great paper showing how strong proprietary AI like #GPT4 can be paired with #OSS LLM to ensure LLM output validity, e.g. valid JSON.
Many devs complain that #LLMs cannot be reliably used in production if the output is not valid, for instance, if one wants to use LLMs to generate SQL queries or JSON, it is crucial that the output is valid.
Frameworks have arisen to constrain the outputs of the LLM to follow some constraints, like outlines (https://github.com/outlines-dev/outlines), but they assume access to logits.
This makes them incompatible with proprietary LLMs like GPT4 that donโt share logits, so one can only use open-source LLMs that are much less performant.
This paper shows how can use powerful proprietary LLMs like GPT4 to create a first unconstrained sketch and refine it using an OSS model like Llama 2 where logits are accessible, to rewrite the sketch following some specific constraints.
They show that GPT4 Precision can be increased by 14% (43% before, 57% after), by boosting it with constrained output on information extraction on Wiki-NRE!
reacted to
philschmid's
post with โค๏ธ
10 months ago
Post
What's the best way to fine-tune open LLMs in 2024? Look no further! ๐ย I am excited to share โHow to Fine-Tune LLMs in 2024 with Hugging Faceโ using the latest research techniques, including Flash Attention, Q-LoRA, OpenAI dataset formats (messages), ChatML, Packing, all built with Hugging Face TRL. ๐
It is created for consumer-size GPUs (24GB) covering the full end-to-end lifecycle with:
๐กDefine and understand use cases for fine-tuning
๐ง๐ปโ๐ปย Setup of the development environment
๐งฎย Create and prepare dataset (OpenAI format)
๐๏ธโโ๏ธย Fine-tune LLM using TRL and the SFTTrainer
๐ฅย Test and evaluate the LLM
๐ย Deploy for production with TGI
๐ย https://www.philschmid.de/fine-tune-llms-in-2024-with-trl
Coming soon: Advanced Guides for multi-GPU/multi-Node full fine-tuning and alignment using DPO & KTO. ๐
It is created for consumer-size GPUs (24GB) covering the full end-to-end lifecycle with:
๐กDefine and understand use cases for fine-tuning
๐ง๐ปโ๐ปย Setup of the development environment
๐งฎย Create and prepare dataset (OpenAI format)
๐๏ธโโ๏ธย Fine-tune LLM using TRL and the SFTTrainer
๐ฅย Test and evaluate the LLM
๐ย Deploy for production with TGI
๐ย https://www.philschmid.de/fine-tune-llms-in-2024-with-trl
Coming soon: Advanced Guides for multi-GPU/multi-Node full fine-tuning and alignment using DPO & KTO. ๐
- 4 replies
reacted to
gsarti's
post with โค๏ธ
10 months ago
Post
๐ฅ Today's pick in Interpretability & Analysis of LMs: Fine-grained Hallucination Detection and Editing For Language Models by
@abhika-m
@akariasai
@vidhisha
et al.
Authors introduce a new taxonomy for fine-grained annotation of hallucinations in LM generations and propose Factuality Verification with Augmented Knowledge (FAVA), a retrieval-augmented LM fine-tuned to detect and edit hallucinations in LM outputs, outperforming ChatGPT and LLama2 Chat on both detection and editing tasks.
๐ Website: https://fine-grained-hallucination.github.io
๐ Paper: Fine-grained Hallucination Detection and Editing for Language Models (2401.06855)
๐ Demo: fava-uw/fava
๐ค Model: fava-uw/fava-model
๐ก Dataset: fava-uw/fava-data
Authors introduce a new taxonomy for fine-grained annotation of hallucinations in LM generations and propose Factuality Verification with Augmented Knowledge (FAVA), a retrieval-augmented LM fine-tuned to detect and edit hallucinations in LM outputs, outperforming ChatGPT and LLama2 Chat on both detection and editing tasks.
๐ Website: https://fine-grained-hallucination.github.io
๐ Paper: Fine-grained Hallucination Detection and Editing for Language Models (2401.06855)
๐ Demo: fava-uw/fava
๐ค Model: fava-uw/fava-model
๐ก Dataset: fava-uw/fava-data
posted
an
update
10 months ago
Post
๐ช32k-context BERT for embedding and RAG on long corpus
Monarch Mixer is a new architecture to enable long context BERT for large corpus and can be fine-tuned for large context retrieval.
Quite interesting and important as BERT is still the most used LLM in production for "old school" tasks like classification, NER, embeddings, but is also a key component for RAG.
Paper: https://arxiv.org/abs/2310.12109
Blog: https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval
GitHub: https://github.com/HazyResearch/m2
Monarch Mixer is a new architecture to enable long context BERT for large corpus and can be fine-tuned for large context retrieval.
Quite interesting and important as BERT is still the most used LLM in production for "old school" tasks like classification, NER, embeddings, but is also a key component for RAG.
Paper: https://arxiv.org/abs/2310.12109
Blog: https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval
GitHub: https://github.com/HazyResearch/m2
- 1 reply