Automate document processing with Form Recognizer and Logic Apps (Part 2)
Tags: azure, ai, cloud, logicapps
Published: 2023-07-10 13:51:50

Processing of forms and documents is part of several scenarios both in business and in everyday life. Manual data extraction from documents, either in electronic or printed format, is time-consuming, costly, and error-prone.

Azure Form Recognizer is an Applied AI Service that enables you to extract text, table data, key-value pairs, and layout information from forms and documents. In this two-part series, you will learn how to build an end-to-end document processing automation solution utilizing Azure Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI.

In the first part, you trained two custom models for extracting key-value pairs from customer service review forms and composed these models together into a single model. In this article, you will build an automated form processing solution.

You will learn how to:
* Create a Logic App workflow that responds to Event Grid events.
* Integrate Form Recognizer into a Logic Apps workflow.
* Store the extracted information from the Form Recognizer’s model in Azure Cosmos DB.
* Use Power BI to visualize the insights from the analysis of the forms.

To build this solution, you will need an Azure subscription. If you don’t have one, you can sign up for an [Azure free account](https://azure.microsoft.com/free/?WT.mc_id=AI-MVP-5004971).


## Scenario details

Consider that you work in a company that provides customer service to a variety of customers. Every day, customers leave reviews about their experiences with the customer service they received. You need to analyze these reviews in order to identify areas of improvement and to track customer satisfaction. These reviews are submitted as paper forms or PDFs and may include up to two different form types. To make this process easier, we need to use a single service to analyze all the reviews.

In the following image, you can see the workflow that we will build to automate the process of extracting and analyzing customer service reviews. In this post, we will focus on the second and third steps of the workflow.

![Form processing automation solution](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/form-processing-solution_hu0d2f9a5c9b70f23615414c7b18aa227f_197712_790x274_fit_q95_h2_box_3.webp)


## Architecture

The following architecture diagram illustrates the main components involved in the automated form processing solution that we are building and the information flow. The system receives the forms (either in PDF or scanned image format), extracts the fields from the form and saves the extracted data in Azure Cosmos DB. Power BI is then used to visualize the insights from the data.

![Architecture diagram that shows an architecture for automating form processing.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/architecture_hucc69a0bc34bf688087dd8b7facbed9fd_48404_790x376_fit_q95_h2_box_3.webp)


### Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

1. PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
2. Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
3. The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
4. The extracted key-value pairs are saved into a NoSQL database in Azure Cosmos DB.
5. Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.


## Components

### Setup Azure Blob Storage

You will create a storage account and a container to upload the forms that will be processed by our Logic App.

1. Sign in to the Azure portal, search for **Storage accounts** and then select **Create**.
2. Create a storage account with the following settings:
   * **Subscription**: Your Azure subscription.
   * **Resource group**: Select an existing resource group (i.e. the resource group that you created in the [Part 1](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-1/)) or create a new one.
   * **Storage account name**: Enter a unique name.
   * **Region**: Choose any available region, for example, **West Europe**.
   * **Performance**: Standard.
   * **Redundancy**: Locally-redundant storage (LRS).
   ![Create a storage account.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/create-storage-account_huc2dfea4d93aea28d1e8f329fe7c12ed6_37817_611x591_fit_q95_h2_box_3.webp)

3. Select **Review + Create** and then select the **Create** button and wait for the deployment to complete. Once the deployment is complete, navigate to your storage account.
4. In the left pane, under **Data storage**, select **Containers**.
5. Create a new container. Select a name and set the public access level to **Container**.
   ![Create a container.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/create-container_hu1ed618d0c66dc493fd9449091123cf31_10835_423x198_fit_q95_h2_box_3.webp)


### Setup Event Grid

Event Grid is a kind of event broker that you can use to integrate applications using events. We will use Event Grid as a trigger to run the Logic App when a file is uploaded in the Azure Storage container.

To use Event Grid, you must first register it as a resource provider in your subscription.

![Register Event Grid as a resource provider in your Azure subscription.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/register-event-grid_hu30b74b4f8419408ca10bcc0dcb0ad87a_103926_1151x525_fit_q95_h2_box_3.webp)


### Setup Azure Cosmos DB

You need to create an Azure Cosmos DB account, database, and container to store the fields extracted from the safety forms.

1. In the Azure portal, search for **Azure Cosmos DB** and then click **Create**.
2. Create a new **Azure Cosmos DB for NoSQL account** by selecting the corresponding card in the **Which API best suits your workload?** window.
3. Create an Azure Cosmos DB account with the settings below:
   * **Subscription**: Your Azure subscription.
   * **Resource group**: Select an existing resource group (i.e. the resource group that you created in the [Part 1](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-1/)) or create a new one.
   * **Account name**: Enter a unique name.
   * **Location**: Choose any available region, for example, **West Europe**.
   * **Capacity mode**: Provisioned throughput.
   * **Limit total account throughput**: selected.
   ![Create an Azure Cosmos DB account.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/create-cosmos-db_hu19bc4d84482b0dd0cebe030849f8616e_50129_702x571_fit_q95_h2_box_3.webp)

4. Select **Review + Create** and then select the **Create** button and wait for the deployment to complete. Once the deployment is complete, navigate to your resource.
5. In the left pane, select **Data Explorer**. Then, create a new database and a container.
   ![Create a database and a container in Azure Cosmos DB.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/create-database-container_hu8e4449d8cc90ca8fd12146f5ec46b704_93042_750x413_fit_q95_h2_box_3.webp)


### Create a Logic App

The last component that we need to provision is a Logic App.

1. In the Azure portal, search for **Logic apps** and then click **+ Add**.
2. Create a Logic App by specifying the following parameters:
   * **Subscription**: Your Azure subscription.
   * **Resource group**: Select an existing resource group (i.e. the resource group that you created in the [Part 1](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-1/)) or create a new one.
   * **Logic App name**: Enter a unique name.
   * **Region**: Choose any available region, for example, **West Europe**.
   * **Plan type**: Consumption.
   * **Zone redundancy**: Disabled.
   ![Create a Logic App.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/create-logic-app_hu1bbd652d55667d3ec83bc727ae14f406_42567_492x513_fit_q95_h2_box_3.webp)

3. Select **Review + Create** and then **Create**.
4. Once the deployment is complete, navigate to your Logic App resource. You are now ready to build the document processing workflow.


## Build the workflow

You will build the document processing workflow using the Logic App Designer, a graphical user interface that enables you to create workflows visually.

1. Once the Logic App resource is created, you will see the starter window of Logic App designer. In the **Start with a common trigger** window, select the **When an Event Grid event occurs** block.
   ![Create a workflow that executes when an Event Grid event occurs.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/new-workflow_hu9dc0bd29036db352209b93fd0d4b3025_152901_800x429_fit_q95_h2_box_3.webp)
   > If you can’t find this block, create an empty workflow and then select the **When a resource event occurs** trigger.

2. Sign in to your account. In the **When a resource event occurs** block specify the following details:
   * **Subscription**: Select your subscription.
   * **Resource Type**: `Microsoft.Storage.StorageAccounts`
   * **Resource Name**: The name of your Storage account.
   * **Event Type Item – 1**: `Microsoft.Storage.BlobCreated`
   ![Setup the When a resource event occurs trigger.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/event-grid-trigger_huef1df4eaf2c9281838a6968d14110931_23650_618x415_fit_q95_h2_box_3.webp)

      > In the Event Grid trigger block, you can optionally apply filters. For example, you can use the **Prefix Filter** to subscribe to events from a specific container in the Storage account or use the **Suffix Filter** to filter events based on the extension of the uploaded file, such as *.jpg* or *.pdf*.

3. Add a new step. In the **Choose an operation** window, search for **Parse JSON** and then select the respective action.

4. In the **Parse JSON** block, enter the following parameters:
   * **Content**: `Body`
   * **Schema**: Use the following schema.
      ```json
       {
           "properties": {
               "data": {
                   "properties": {
                       "api": {
                           "type": "string"
                       },
                       "blobType": {
                           "type": "string"
                       },
                       "clientRequestId": {
                           "type": "string"
                       },
                       "contentLength": {
                           "type": "integer"
                       },
                       "contentType": {
                           "type": "string"
                       },
                       "eTag": {
                           "type": "string"
                       },
                       "requestId": {
                           "type": "string"
                       },
                       "sequencer": {
                           "type": "string"
                       },
                       "storageDiagnostics": {
                           "properties": {
                               "batchId": {
                                   "type": "string"
                               }
                           },
                           "type": "object"
                       },
                       "url": {
                           "type": "string"
                       }
                   },
                   "type": "object"
               },
               "dataVersion": {
                   "type": "string"
               },
               "eventTime": {
                   "type": "string"
               },
               "eventType": {
                   "type": "string"
               },
               "id": {
                   "type": "string"
               },
               "metadataVersion": {
                   "type": "string"
               },
               "subject": {
                   "type": "string"
               },
               "topic": {
                   "type": "string"
               }
           },
           "type": "object"
       }
      ```
      ![Configure the Parse JSON block.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/parse-json_hufba8241a3992c74aca2299d4cb465c5e_74813_857x469_fit_q95_h2_box_3.webp)

5. Find the **Analyze Document for Prebuilt or Custom models (v3.0 API)** in the Form Recognizer connector and add it to your workflow.
   ![Add an action to analyze a form using Form Recognizer.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/form-recognizer-api_hu9b76030f831d4fa703c20987a63c2a78_64829_667x451_fit_q95_h2_box_3.webp)

6. Connect to Form Recognizer:
   * **Connection name**: Enter a name for the connection.
   * **Endpoint URL**: Enter the **Endpoint URL** of your Form Recognizer resource.
   * **Account Key**: Enter the **Key 1** of your Form Recognizer resource.
   
      > You can find the **Endpoint URL** and the **Key** in your Form Recognizer resource, under **Keys and Endpoint**.

7. Then, specify the following parameters:
   * **Model Identifier**: Enter the name of the model that will be used to analyze the forms (i.e. the name of the composed model that you created in [Part 1](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-1/)).
   * **Document/Image URL**: Click **Add dynamic content** and select the `url`.
   ![Configure the Form Recognizer action.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/form-recognizer-action_hudd7d97b33d6bda57aad10deab78435c1_17213_616x315_fit_q95_h2_box_3.webp)

### Save the extracted data

The JSON response of the **Analyze Document for Prebuilt or Custom models (v3.0 API)** action includes several information, such as the text extracted from the form, layout information, selection marks, and key-value pairs. We are interested in extracting the fields and their associated values. In the following steps, you will build a pipeline that extracts the fields from the JSON response, processes the values, generates a JSON file and saves the file in Azure Cosmos DB.

1. Add a **Parse JSON** action and specify the following details:
   * **Content**: Select **Add dynamic content** and find the block called `fields`.
   * **Schema**: To generate the schema, use a sample JSON response of the Form Recognizer action. Click **Use sample payload to generate schema** and paste the section of the Form Recognizer’s response identified by the term `"fields"`.
   ![Use a Parse JSON action to extract key-value pairs.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/parse-fields_hu16ba4c538942a2c4a6a4db7ba40c485e_29196_566x443_fit_q95_h2_box_3.webp)
   
      > The Logic App automatically adds a **For each** block around the **Parse JSON** block.

2. Add a **Compose** action (inside the **For each** block). We will save the extracted fields in the following format:
      ```json
    {
      "BillingPayment": "value1",
      "Comments": "value2",
      "OtherService": "value3",
      "ProductServiceInformation": "value4",
      "Recommend": "value5",
      "Satisfaction": "value6",
      "Suggestions": "value7",
      "TechnicalSupport": "value8",
      "id": "id"
    }
      ```

3. To extract the value associated with each field, use `if` statements. You can add an `if` statement by selecting **Add dynamic content > Expression**.

    For example, to extract the values associated with the fields `BillingPayment`, `Comments` and `Recommend`, you can use the below expressions:

    ```javascript
    // BillingPayment
    if(and(contains(body('Parse_Fields')?['BillingPayment'], 'content'), equals(body('Parse_Fields')?['BillingPayment']?['valueSelectionMark'], 'selected')), 1, 0)

    // Comments
    if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], 'NA')

    // Recommend
    if(and(contains(body('Parse_Fields')?['RecommendVeryLikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryLikely']?['valueSelectionMark'], 'selected')), 'Very likely', if(and(contains(body('Parse_Fields')?['RecommendLikely'], 'content'), equals(body('Parse_Fields')?['RecommendLikely']?['valueSelectionMark'], 'selected')), 'Likely', if(and(contains(body('Parse_Fields')?['RecommendNeutral'], 'content'), equals(body('Parse_Fields')?['RecommendNeutral']?['valueSelectionMark'], 'selected')), 'Neutral', if(and(contains(body('Parse_Fields')?['RecommendUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendUnlikely']?['valueSelectionMark'], 'selected')), 'Unlikely', if(and(contains(body('Parse_Fields')?['RecommendVeryUnlikely'], 'content'), equals(body('Parse_Fields')?['RecommendVeryUnlikely']?['valueSelectionMark'], 'selected')), 'Very unlikely', 'NA')))))
    ```

    In a similar way, you can extract the values corresponding to the remaining fields.
    ![Generate JSON output.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/compose-action_hu39e0f22b8e60e8fabadebe92712c7557_19921_554x426_fit_q95_h2_box_3.webp)

4. Click **Add an action** and select the **Create or update document (V3)** action of the Azure Cosmos DB connector. Then, configure the Azure Cosmos DB connection by specifying the values below:
   * **Connection name**: Enter a name for the connection.
   * **Authentication type**: Access Key.
   * **Access Key to your Cosmos DB account**: Enter the primary (or secondary) key of your Azure Cosmos DB resource.
   * **Account ID**: Enter the name of your Azure Cosmos DB account.
   ![Configure the Azure Cosmos DB connection.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/cosmos-db-connection_hue3080d022f122fb99859203f6161ef56_51637_602x490_fit_q95_h2_box_3.webp)
   
      > To find the primary key of your Cosmos DB account, navigate to your Cosmos DB resource and select **Keys**.

5. In the **Create or update document (V3)** block, specify the ID of your database and collection. Then, under **Document**, select **Add dynamic content** and add the block named `Outputs`.
   ![Save the results in an Azure Cosmos DB database.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/save-results-cosmos-db_hu9a13a8bb8e93893a1de323f4706fa41a_17749_586x382_fit_q95_h2_box_3.webp)

Save the workflow and then click **Run Trigger > Run**. Upload a file to your Azure Storage container to test the Logic App.


## Connect Power BI to Azure Cosmos DB

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

1. Open Power BI Desktop and in the **Home** tab select **Get data > More…**
2. Choose the **Azure Cosmos DB v1** connection and click **Connect**.
3. In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click **OK**.
4. Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the extracted data.

![Power BI dashboard.](https://sfoteini.github.io/images/post/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/power-bi-dashboard_hud4e64dcb0aba8e1d1e5189f560b55220_50118_985x549_fit_q95_h2_box_3.webp)


## Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB and Power BI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by adding AI capabilities (such as, sentiment analysis, key-phrase extraction, opinion mining) in your Logic App workflow.

Here are some additional scenarios that leverage Azure Form Recognizer:

* [Use Azure Functions and Python to process stored documents – Microsoft Docs](https://learn.microsoft.com/azure/applied-ai-services/form-recognizer/tutorial-azure-function?view=form-recog-3.0.0&WT.mc_id=AI-MVP-5004971)
* [Create Form Recognizer workflows with AI Builder – Microsoft Docs](https://learn.microsoft.com/ai-builder/create-form-processing-model?toc=%2Fazure%2Fapplied-ai-services%2Fform-recognizer%2Ftoc.json&bc=%2Fazure%2Fapplied-ai-services%2Fform-recognizer%2Fbreadcrumb%2Ftoc.json&view=form-recog-3.0.0&WT.mc_id=AI-MVP-5004971)


## Clean-up

If you have finished learning, you can delete the resource group from your Azure subscription:

1. In the Azure portal, select **Resource groups** on the right menu and then select the resource group that you have created.
2. Click **Delete resource group**.

---

👋 **Hi, I am Foteini Savvidou!**
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 [LinkedIn](https://www.linkedin.com/in/foteini-savvidou) | [Blog](https://sfoteini.github.io/) | [GitHub](https://github.com/sfoteini)