How to build an AI data assistant that actually works
Published: 2023-10-17 16:40:13

_This was originally posted [here](www.buster.so/post/how-to-build-an-ai-data-assistant-that-actually-works)_

## Lessons we’ve learned while building [Buster](https://buster.so).
AI Data Assistants have been all the rage over the last 6 months. It feels like one of the most practical use cases of Generative AI.  With the open source libraries available, you could build a data assistant in a weekend.  However, most of the time the results are disappointing in a real-world scenario.  We’ve spent the last 6 months building one that actually works and heres what we learned.

## Context Is Everything

Just like we wouldn’t expect a junior analyst on their first day of work to deliver accurate insights, we can’t expect an LLM to do the same. Our initial iteration of Buster was basically us cramming table schemas into an LLM and asking it to translate our question to SQL.  As you can imagine, we had less than stellar results.  Buster seemed to lack context to understand our questions.

The reality is, there is a lot of context that analysts use while fielding requests within an organization.  For an analyst, that context is found in written documentation, gleaned from some diagrams, discovered while working on requests, or transferred from one analyst to the next.

So the challenge is, how do you gather that context in an efficient manner. Some of it can be done during the onboarding of a dataset (i.e. column descriptions, table keywords, enumeration values, etc.).  This part is pretty straightforward and we’ve found that an LLM can generate about 90% of the context you need around tables and columns. We typically refer to this step as ‘Documenting your Database’.

After pushing this out, we realized that while database documentation significantly increased performance, there is still a lot of undocumented context that has to be learned over time.  It’s basically impossible for a data team to unload years worth of context into an LLM.  Especially as data changes over time. For LLM-powered analytics to be successful, there needs to be continuous learning and context collection.

After doing this successfully, we started to see Buster hit 98%+ accuracy across queries.

## Utilizing Retrieval Augmented Generation (RAG)

The next problem we ran into is that LLMs are incapable of handling enormous context lengths while still producing accurate and reliable code.  As we tried to cram all the tables, columns, and context in between, we saw performance decrease on generated queries.  Not only that, but our application became extremely expensive and latent.

To combat these problems, we built an intermediate orchestration layer that allows us to deliver to the LLM exactly what it needs to answer the user question.  What I mean by this is that instead of cramming 100 columns, 500 possible values, and any other context into an LLM, we tell the LLM specifically which tables, columns, relationships, and any other context to use in order to answer the user question.

A huge piece of this is our vector search and retrieval layer.  We learned that we could vectorize a majority of the context we were collecting then pair that with some creative logic on our end to deliver dynamic context to the LLM.  Retrieval from a vector store isn’t a silver bullet for this, so we had to build out validation and ranking logic to be able to deliver the most relevant and valid context to the LLM.

By controlling the context on our end, we were able to reduce latency and cost while providing a better experience to our users.

## Explainability and Flexibility

We learned that users were quick to lose trust in Buster if they got back a wrong or unnatural result.  The honest truth is that LLMs will produce unexpected results.  So, the challenge is two-fold. First, how do you help a user validate their results?  This isn’t so straightforward as showing them the generated SQL.  Depending on your end user, they may not even know SQL. Second, how do you build your application to be flexible enough for them to fix or refine things?  You have to have workflows for when something doesn’t go as planned and it has to be easy to do.

To solve this, we decided to build Buster for natural language dialogue.  Instead of asking a question and getting SQL + Results back, we built Buster to be a chat space, almost as if you were conversing with an analyst via Slack or email.  The key that we found was to continuously explain to the user how the data was being gathered and assumptions (if any) that were being made about the dataset.  This enabled users to validate results that they got back from Buster and then continue to refine or fix them on their own.

An interesting challenge we saw on the flexibility side of things was that a user’s follow-up question could actually change the original question that was asked.  While the questions themselves may be similar, the SQL needed to answer them could be completely different.  We had to modify our orchestration layer to account for how metrics and reports could change over the course of a user chat. Realistically a user could ask a vague question that requires one piece of context, then follow up and require completely different context.

## Wow this post feels long…

Given that this post is already really long, we’ll plan on releasing more content around other learnings we’ve had.  Stick around for more content as we talk about how we view security, how we design a UI for this, and how crucial it is to provide management tools to data teams.

In the meantime, if you or your organization would like to use Buster. [Request a demo here](https://buster.so).

Additionally, follow this link if you want to test out our [interactive demo](https://app.buster.so/demo).