Azure Cognitive Search - Searching documents with Blob Storage
Tags: azure, search, cosmos, blob
Published: 2023-10-02 07:31:08

We will setup a complete flow on how we can use both Cosmos DB and Blob storage together in Azure Cognitive Search.
Azure Cognitive Search is a powerful tool which can help in searching documents across both Cosmos and Blob based on various parameters. 
Cosmos DB provides the parameters of the file which can be helpful in searching like extension, filename, author.
Blob Storage, where the actual files are present, can be used in Search when we have to search the document based on the content in the document. 
The terms which we will be using in this post will be referenced from this [post](https://dev.to/ishwar398/getting-started-with-azure-cognitive-search-1f9e) which talks about the theoretical details of Azure Search.

## Setting up the data sources (assuming Blob and Cosmos are already created)
From the left pane in the Azure Cognitive Search, select the data sources option. Click on the **Add Data Source** button.

![Adding a data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w34l5ok6wazqedfxatxk.png)

Cognitive Search gives us option to select from a variety of sources.

![Selecting the data source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hkxwpl78ahbvw2tgmnxd.png)

For this post, we will be selecting the Azure blob storage. On selecting Azure blob storage, it will ask for selecting the 'Connection string'. There an option below to select from existing connection. 
On clicking this, you'll get the list of Blob storages in the subscription. Select the blob storage and then select the container.

![Select the container](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iitj5zx3chepti2vagp6.png)

Once you select the instance, the connection string will be automatically populated. Provide a name to it.
If you need to read the files from a specific folder in the container, you can add the path in the 'Blob Folder' box.

![Blob Folder](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8w2rkqhkg1egmr3620jp.png)

On the Search's data source section, you'll be able to see the data source.
Now click on **Indexes** option, and click on **Add Index**.
Then we need to add the fields on which we will be making a search.
By default, when we select Blob storage as Data Source, the indexer is able to perform an OCR on the text documents (PDF, Word, RTF etc.) and add the content of the document in the search index. To make the document searchable by the content within them, you'll need to add the **content** field in the index and mark it as Searchable. 
This will be a string search and there needs to be an Analyzer selected for it. 
Analyzer defines your searching strategy, how your search query will be parsed, how the AND, OR, NOT will be handled. For this example, we will be using the **Standard Lucene** parser.

![Adding Index](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nmysrr4rfhpu3qez1a8c.png)

Once the index is created, it will be shown in the Index list.

![Index](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ofp0m4kqsfkkg4r95ucz.png)

Now we will start creating the Indexer. Indexer is the link between the Data Source and the Index. 

Click on **Indexers** and then click on **Add Indexer**.
Fill the basic details.

1. Give a proper name to the indexer
2. Select the target index where the indexer will push the data
3. Choose the data source from where the indexer will pull the data

![Indexer - Basic details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8llk8fzqyutj9dglwd2i.png)

4. Skills is something we will see in a different post, so for now skip it.
5. Select the schedule when the indexer should run and fetch the data from the data source.

In the Advanced settings, you can set which data types you want to include or exclude. Also, add enrichments etc.

## Indexer in action
The data source configured has 2 documents in it.

![Files in blob storage container configured as source](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44sqi9xb0jv7gj3105mi.png)

When we run the indexer, it fetches the documents in this container.

![Documents fetched from the blob storage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oby4otd5rc78d7128vss.png)

Same document count we'll be able to see on the index as well since the indexer has pushed the data to the target index.

![Document count on Index](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/trqdvwc2llanmejyhzxp.png)

## Search in action

The document uploaded to the search is as follows:

![Test document uploaded](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2wamqf0jgxy2o5u5ejdm.png)

By default, the Azure cognitive search provides OCR on the text documents (PDF, DOCX, RTF etc.), so it is able to fetch the content from the PDF file we uploaded and index the content as well. 
So we if try to search for anything from the content, it brings back the file as result.


![Search Results when we search for any word from the document](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o2lfbu55eop2jcsysfoq.png)