How to add flexibility to your RAG applications by choosing the right configuration(s)
Tags: machinelearning, generativeai
Published: 2024-06-04 04:14:22

[Knowledge Bases for Amazon Bedrock](https://aws.amazon.com/bedrock/knowledge-bases/) is a fully managed capability that helps you implement the entire RAG workflow from ingestion to retrieval and prompt augmentation without having to build custom integrations to data sources and manage data flows.

There are several configurations you can tweak to customize retrieval and response generation. This done via query configuration parameters which can be applied via the console, API or the SDK. 

Let's walk through them one by one.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jr7v33r7mg72rm503nvn.png)

## Maximum number of retrieved results

Semantic search (the retrieval in RAG) are usually Top-K searches i.e. *"Give me the best K search results in response to my query"*. By default Amazon Bedrock returns up to five results in the response. But you can modify this:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bvb0vakbm4mvjx6jwmq2.png)

## Search Type

You can actually decide to combine semantic search with the "good old" text based search - Choose the **Hybrid** search type if that's the case. Combines searching vector embeddings (semantic search) with searching through the raw text. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k81ovy09sec15pdg9ccw.png)

Opting for the **Semantic** option only searches through the vector embeddings.

> Note: At the time of writing **Hybrid** search is currently only supported for Amazon OpenSearch Serverless vector stores that contain a filterable text field. Amazon Bedrock falls back to using semantic search if you configure a different vector store or your Amazon OpenSearch Serverless vector store doesn't contain a filterable text field.

## Prompt template

The **"A"** (Augmented) in RAG is when the search results are combined with the prompt. Amazon Bedrock uses a default prompt template. But you can do further prompt engineering using prompt placeholders (such as `$query$`, `$search_results$`, etc.).

Prompt templates differ based on the chose model. For example, here is the one for [Amazon Titan Text Premier](https://aws.amazon.com/bedrock/titan/):

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6qkiyrgcawatmozqefvb.png)

... and here is the one for **Claude Haiku**:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m010u2aadw7ranv2zix5.png)

> Note: This is only use with [RetrieveAndGenerate](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html) API

## Inference parameters 

These are values that you can adjust in order to influence the model response. This includes `temperature`, `topP`, `topK`, stop sequences, etc.

You can set these with Knowledge Base RAG queries as well.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/djrpodst3r6enutd45e9.png)

> Note: This is only use with [RetrieveAndGenerate](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html) API

## Guardrails

With [Guardrails in Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html), you can implement safeguards for your generative AI applications based on your use cases and responsible AI policies. A guardrail consists of multiple policies to avoid content that falls into undesirable or harmful categories.

Once you create a Guardrail, simple associate it with the knowledge base:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qmultmcb3ypss1szv0gh.png)

> Note: This is only use with [RetrieveAndGenerate](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html) API

## Metadata files

Retrieval does not have to be just limited based on the semantic search results. You can further tune queries by including additional **metadata** files with your source documents. It can contain attributes as key-value pairs that you define for a source document.

You can use filter (equals, greater than, etc.) and logical (and, or) search operators along for metadata based filters.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gldl49si7esa0omqrtab.png)

> For details, you can refer to [Add metadata to your files to allow for filtering](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html#kb-ds-metadata)


## Bonus - Chunking and Delete policy

> Strictly speaking, these are not query configurations, but definitely worth knowing

- **Chunking:** During data ingestion (from source to the chosen vector database), the each file is split into chunks using one of the following strategies - no chunking (each file = a chunk), default (each chunk = ~300 tokens), fixed size (you define the size)
- **Data Deletion Policy:** The default policy is `DELETE`, which means that the underlying vector will be deleted along with the knowledge base. To change prevent the vector store deletion, change the policy to `RETAIN`.

## Conclusion

I showed examples for AWS console, but like I mentioned earlier, these are applicable to the SDK and API as well. For example, [here is how the RetrieveAndGenerate API uses these](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_GenerationConfiguration.html) configuration parameters. 

Read more in [Query configurations](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html). Happy building!