Adding an Amazon Bedrock Knowledge Base to the Forex Rate Assistant
Tags: aws, ai
Published: 2024-05-17 05:38:09

## Introduction

In our journey of experimenting with Amazon Bedrock up to this point, we have built a [basic forex assistant](https://dev.to/aws-builders/building-a-basic-forex-rate-assistant-using-agents-for-amazon-bedrock-17fp) as the basis for further enhancements to evaluate various Bedrock features and generative AI (gen AI) techniques. Our next step is to integrate a knowledge base to the agent, so that it can provide information about foreign currency exchange in general.

In this blog post, we will define a representative use case for a RAG scenario for the forex rate agent, build a forex knowledge base, and attach it to the agent. Accuracy and performance of a gen AI application is also essential, so we'll conduct some tests and discuss challenges associated with RAG workflows.

## About Knowledge Bases for Amazon Bedrock

[Knowledge Bases for Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html) is a service that provides managed capability for implementing [Retrieval Augmented Generation (RAG)](https://aws.amazon.com/what-is/retrieval-augmented-generation/) workflows. Knowledge bases can be integrated with Bedrock agents to seamlessly enable RAG functionality, or be used as a component in custom AI-enabled applications using API.

Knowledge Bases for Amazon Bedrock automates the ingestion of source documents, by generating embeddings with a foundation model, such as [Amazon Titan](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html) or [Cohere Embed](https://aws.amazon.com/bedrock/cohere-command-embed/), and storing them in a supported vector store as depicted in the following diagram:

![Bedrock knowledge base data pre-processing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w1e9tdq6x7xzqg95ur70.png)

To keep things simple, the service provides a quick start option that provisions on your behalf an [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html) vector database for its use.

Aside from [native integration into a Bedrock agent](https://docs.aws.amazon.com/bedrock/latest/userguide/agents-kb-add.html), the [Agents for Amazon Bedrock Runtime API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Agents_for_Amazon_Bedrock_Runtime.html) offers both the ability to perform raw text and semantic search, and the ability to retrieve and generate a response with a foundation model, on knowledge bases. The latter allows the community to provide tighter integration in frameworks such as [LangChain](https://js.langchain.com/docs/integrations/retrievers/bedrock-knowledge-bases) and [LlamaIndex](https://docs.llamaindex.ai/en/latest/examples/retrievers/bedrock_retriever/) to simplify RAG scenarios. The runtime flow is shown in this diagram:

![Bedrock knowledge base runtime execution](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0mizx0v7z7ppi2z6jmko.png)

## Enhancing the forex rate assistant use case

In the blog post [Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock](https://dev.to/aws-builders/building-a-basic-forex-rate-assistant-using-agents-for-amazon-bedrock-17fp), we created a basic forex assistant that helps users look up the latest forex rates. It would be helpful if the assistant can also answer other questions on the broader topic.

While information about the history of forex would be useful, the Claude models already possess such knowledge so it would not make a great use case for knowledge bases. As I searched for more specific subtopics, I found the [FX Global Code](https://www.globalfxc.org/fx_global_code.htm), a set of common guidelines developed by the [Global Foreign Exchange Committee (GFXC)](https://www.globalfxc.org/overview.htm) which establishes universal principles to uphold integrity and ensure the effective operation of the wholesale FX market. The FX Global Code is conveniently available in [PDF format](https://www.globalfxc.org/docs/fx_global.pdf), which is perfect for ingestion by the knowledge base.

## Requesting model access and creating the S3 bucket for document ingestion

Let's start with the [prerequisites](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-prereq.html) by [requesting for model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html). For the forex knowledge base, we will be using the **Titan Embeddings G1 - Text** model. You can review the model pricing information [here](https://aws.amazon.com/bedrock/pricing/).

Next, we need to create the S3 bucket from which the knowledge base will ingest source documents. We can quickly do so in the S3 Console with the following settings:

- **Bucket name:** forex-kb-*&lt;region&gt;*\-*&lt;account\_id&gt;* (such as `forex-kb-use1-123456789012`)
- **Block all public access:** Checked (by default)
- **Bucket versioning:** Enable
- **Default encryption:** SSE-S3 (by default)
    

Once the S3 bucket is created, download the [FX Global Code PDF file](https://www.globalfxc.org/docs/fx_global.pdf) and upload it to the bucket:

![S3 bucket and source document for the knowledge base](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dq3fto273rzud4mxvq1k.png)

This is sufficient for our purpose. For more information on other supported document formats and adding metadata for the filtering feature, refer to the [Amazon Bedrock user guide](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html).

## Creating a knowledge base along with a vector database

Next, we can create the knowledge base in the Amazon Bedrock console following the steps below:

1. Select **Knowledge bases** in the left menu.
    
2. On the **Knowledge bases** page, click **Create knowledge base**.
    
3. In **Step 1** of the **Create knowledge base** wizard, enter the following information and click **Next**:
    - **Knowledge base name:** ForexKB
    - **Knowledge base description:** A knowledge base with information on foreign currency exchange.
    ![Create knowledge base - step 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ljsf0knqqvo9m9titsq3.png)
    
4. In **Step 2** of the wizard, enter the following information and click **Next**:
    - **Data source name:** ForexKBDataSource
    - **S3 URI:** *Browse and select the S3 bucket that we created earlier*
    ![Create knowledge base - step 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8mzt40k4h277h73uo6m5.png)
    
5. In **Step 3** of the wizard, enter the following information and click **Next:**
    - **Embeddings model:** Titan Embeddings G1 - Text v1.2
    - **Vector database:** Quick create a new vector store
    ![Create knowledge base - step 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/clcf7ce3adv8thki3pxk.png)
    
6. In **Step 4** of the wizard, click **Create knowledge base**.
    

The knowledge base and the vector database, which is an Amazon OpenSearch Server collection, will take a few minutes to create. When they are ready, you'll be directed to the knowledge base page, where you will be prompted to synchronize sync the data source. To do so, scroll down to the **Data source section**, select the radio button beside the data source name, and click **Sync**:


![Sync data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lc86uzbx7e7lehz7qmrj.png)

It will take less than a minute to complete, since we only have a single moderately-sized PDF document. Now the knowledge base is ready for some validation.

## Testing the knowledge base

A knowledge base must provide accurate results, so we need to validate it against information we know exists out of the source documents. This can be done using the integrated test pane. To simulate an end-to-end test for the RAG scenario, configure the test environment in the Bedrock console as follows:

1. Enable the **Generate response** option (which should already be enabled by default).
    
2. Click **Select model**. In the new dialog, select the **Claude 3 Haiku** model and click **Apply**.
    
3. Click on the button with the three sliders, which opens the **Configuration** page. This should expand the test pane so you have more screen real estate to work with.
    

I've prepared a couple of questions after skimming through the FX Global Code PDF file. Let's start by asking a basic question:

> What is the FX Global Code?

The knowledge base responded with an answer that's consistent with the text on page 3 of the document.

![Search result for a basic question](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lsmzeapselu6krxed0fc.png)

To see the underlying search results which the model used to generate the response, click **Show source details**. Similar to agent trace, we can view the source chunks that are related to our question, and the associated raw text and metadata (which is mainly the citation information). Some source chunks refer to the table of contents, which some refers to the same passage from page 3 of the document.

![Source details with raw text and metadata](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d79fywxcm6auti0xe9pn.png)

Next, let's ask something more specific, namely to give me information on a specific principle such as principle 15 which is on page 33 of the PDF file:

![Principle 15 from page 33 of the FX Global Code PDF file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3opiai54sa8fe7dm86lc.png)

> What is principle 15 in the FX Global Code?

Interestingly, the knowledge base doesn't seem to know the answer:

![No result from question about principle 15 in the FX Global Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hvcxks8o41jjpkup5wtq.png)

If I force the knowledge base to use [hybrid search](https://aws.amazon.com/about-aws/whats-new/2024/03/knowledge-bases-amazon-bedrock-hybrid-search/), which combines both sematic and text search for better response, some source chucks were fetched but it does not seem to include one with the text from page 33.

![Search results with hybrid search](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ern2udzb5ycy8eodbzbj.png)

Since there are exactly five results, I figured that it might be limited by the maximum number. After increasing it to an arbitrary 20, the knowledge base finally returned a good response with the default search option:

![Better response after setting maximum number of retrieved results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pw19u6im2xjdydmg8923.png)

This goes to show that just like agents, knowledge bases must be tested and fine-tuned extensively to improve accuracy. The embedding model as well as the underlying vector store may also play a part in the overall behavior of the knowledge base.

In any case, you've successfully created a knowledge base using Knowledge Bases for Amazon Bedrock, which can be integrated with your gen AI applications. To complete our experimentation, let's now integrate it with our forex rate assistant.

## Integrating the knowledge base to the forex rate assistant

If you haven't already done so, please follow the blog post [Building a Basic Forex Rate Assistant Using Agents for Amazon Bedrock](https://dev.to/aws-builders/building-a-basic-forex-rate-assistant-using-agents-for-amazon-bedrock-17fp) to create the forex rate assistant manually, or use the Terraform configuration from the blog post [How To Manage an Amazon Bedrock Agent Using Terraform](https://dev.to/aws-builders/how-to-manage-an-amazon-bedrock-agent-using-terraform-1lag) to deploy it.

Once the agent is ready, we can [associate the knowledge base](https://docs.aws.amazon.com/bedrock/latest/userguide/agents-kb-add.html) to it using the steps below in the Bedrock Console:

1. Select **Agents** in the left menu.
    
2. On the **Agents** page, click **ForexAssistant** to open it.
    
3. On the agent page, click **Edit in Agent Builder**.
    
4. On the **Agent builder** page, scroll down to the **Knowledge bases** section and click **Add**.
    
5. On the **Add knowledge base** page, enter the following information and click **Add:**
    - **Select knowledge base:** ForexKB
    - **Knowledge base instructions for Agent:** Use this knowledge base to retrieve information on foreign currency exchange, such as the FX Global Code.
    ![Add knowledge base](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uf4xds7s3rttux00quyb.png)
    
6. Click **Save and exit**.
    

Once the knowledge base is added, prepare the agent and ask the same first question as before to validate the integration:

> What is the FX Global Code?

The agent responded with a decent answer. In the trace, we can see that the agent invoked the knowledge base as part of its toolset and retrieved the results for its use.


![Agent performing knowledge base search](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gjzgfqczaege0wtsp3e2.png)

We also want to ask the agent to fetch some exchange rate to ensure that the functionality is still working:

> What is the exchange rate from EUR to CAD?

The agent responded with the rate fetched from the `ForexAPI` action group, which is what we expected.

![Agent fetching forex rate as expected](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9adq4b61m0qnxbldhber.png)

However, we run into issues when asking the second question from before:

> What is principle 15 in the FX Global Code?

The agent responded with the inferior answer since we did not adjust the maximum number of retrieval results for the knowledge base.

![Undesirable search results as before](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/al6ndiwqjtsr6jpm29n5.png)

Unfortunately, our issue now is that there is no way that I know of to provide the knowledge base configuration in the agent, so we are stuck. At this point, there's nothing we can do other than opening an AWS support case to inquire about the lack of support... That being said, another angle to look at the problem is that the quality of the source could also affect the knowledge base's search accuracy, which brings us to the topic of common RAG challenges.

## Common RAG challenges

Let's examine the source chunk from the correct answer for the "principle 15" question from our knowledge base test:

![Source chunk containing principle 15 in the FX Global Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/akgua410ys2k888kdgx1.png)

This is also the text that was extracted from the PDF for embedding. Comparing it to the corresponding page in the PDF file, notice the following:

1. The chunk text includes information such as headers and "breadcrumbs" that are not related to the main content.
    
2. The text does not capture the context of the elements in the passage, such as the principle title in the red box and the principle summary in italic.

![How the PDF page looks like](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8y8s8ddw55jcs7qmo7uy.png)

It's fair to think that undesirable artifacts and lack of structural context would impact search accuracy, performance, and ultimately cost. Consequently, it makes sense to perform some data pre-processing before passing the source documents to the RAG workflow. Third-party APIs and tools, such as [LlamaParse](https://github.com/run-llama/llama_parse) and [LayoutPDFReader](https://www.llamaindex.ai/blog/mastering-pdfs-extracting-sections-headings-paragraphs-and-tables-with-cutting-edge-parser-faea18870125), can help with pre-processing PDF data, however keep in mind that source documents may take any forms and there is no one-size-fits-all solution. You may have to resort to developing custom processes for pre-processing and search your unique data.

There are other [challenges in building RAG-based LLM applications](https://datasciencedojo.com/blog/challenges-in-rag-based-llm-applications/) and proposed solutions which you should be aware of. However, some of them cannot be implemented in a managed solution such as Knowledge Bases for Amazon Bedrock, in which case you may need to build a custom solution yourself if you have a genuine need to address them. Such is the eternal quest to balance between effort and quality.

## Don't forget to delete the OpenSearch Serverless collection

Be aware that Knowledge Bases for Amazon Bedrock does not delete the vector database for you. Since the OpenSearch Serverless collection consumes at least one OpenSearch Compute Unit (OCU) which is [charged by the hour](https://aws.amazon.com/opensearch-service/pricing/), you will incur a running cost for as long as the collection exists. Consequently, ensure that you manually [delete the collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-manage.html#serverless-delete) after you have deleted the knowledge base and other associated artifacts. 

## Summary

In this blog post, we created a knowledge base using Knowledge Bases for Amazon Bedrock which we integrate into the [forex rate assistant](https://dev.to/aws-builders/building-a-basic-forex-rate-assistant-using-agents-for-amazon-bedrock-17fp) to allow it to answer questions about the FX Global Code. Through some testing of the solution, we experienced some common challenges for RAG solutions and potential mitigation strategies. Although some of them are not applicable to Bedrock knowledge bases since it abstracts the implementation details, thus highlighting the potential need for a custom solution for more demanding scenarios.

My next step is to enhance the [Terraform configuration for the forex rate assistant](https://dev.to/aws-builders/how-to-manage-an-amazon-bedrock-agent-using-terraform-1lag) to provision and integrate the knowledge base, and to enhance the [Streamlit test app](https://dev.to/aws-builders/developing-a-generic-streamlit-ui-to-test-amazon-bedrock-agents-5c7o) to display citations from knowledge base searches. Be sure to follow the [Avangards Blog](https://blog.avangards.io/) as I continue my journey on building gen AI applications using Amazon Bedrock and other AWS services. Thanks for reading and stay curious!