Automate customer reviews processing with Form Recognizer and Azure OpenAI
Tags: azure, ai, openai, cloud
Published: 2023-07-10 16:07:58

Organizations of all types rely on automated document processing and data extraction to streamline their operations. This article outlines a potential solution for automating PDF form processing, utilizing Azure Form Recognizer for data extraction and Azure OpenAI for "intelligent" data enrichment.

Data enrichment uses Artificial Intelligence (AI) to extract information, uncover patterns, and gain a deeper understanding of the data. This can be achieved through techniques such as key phrase extraction, named entity recognition, sentiment analysis, opinion mining, and custom models to identify essential information. To enrich the data, you can use the pretrained models of the Azure Cognitive Services for Language or train and deploy a custom model in Azure Machine Learning. In this post, I’ll demonstrate how to utilize the Davinci model of the Azure OpenAI Service to perform sentiment analysis and extract key phrases from customer service review forms.

In the two-part series "Automate document processing with Form Recognizer and Logic Apps" you learned how to train custom models in Azure Form Recognizer for extracting key-value pairs from documents and build an end-to-end form processing solution using Form Recognizer, Logic Apps, Azure Cosmos DB, and Power BI.

* [Part 1: Create a custom template model in Azure Form Recognizer](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-1/)
* [Part 2: End-to-end document processing automation solution](https://sfoteini.github.io/blog/automate-document-processing-with-form-recognizer-and-logic-apps-part-2/)

In this article, we will extend this pipeline by incorporating the Azure OpenAI service to enrich the extracted data. You will learn how to:

* Create an Azure OpenAI resource and deploy a model.
* Build a pipeline in Logic Apps to post a request to the Azure OpenAI endpoint, receive and evaluate the response.

> Access to the Azure OpenAI service is currently limited. You can request access to the service by filling out the [application form](https://aka.ms/oaiapply).


## Architecture

The following architecture diagram illustrates the main components involved in the "intelligent" form processing solution that we are building and the information flow. In this post, we will focus on how to integrate the Azure OpenAI service into the pipeline that was built in the previous articles.

![Architecture diagram that shows an architecture for automating form processing.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/architecture_hu68eaf647221ca6aaba16f880c08546c1_57712_790x401_fit_q95_h2_box_3.webp)

### Dataflow

The information flow corresponding to the above architecture diagram is described as follows:

1. PDF forms or scanned images are (manually or programmatically) uploaded to a container in Azure Storage Account.
2. Whenever a form is uploaded to the specified container, an Event Grid event will trigger the Logic App to start processing the form.
3. The Logic App sends the URL of the file to the Form Recognizer API and receives the response data.
4. The Logic App posts a request to the Azure OpenAI service to analyze the sentiment and extract key phrases from the customer's review, and then receives the response data.
5. The extracted data are saved into a NoSQL database in Azure Cosmos DB.
6. Power BI is connected to Azure Cosmos DB to ingest the extracted data and provide dashboards.


## Create an Azure OpenAI resource

You will create an Azure OpenAI resource through the Azure portal.

1. Sign in to the Azure portal and search for **OpenAI**.
2. In the Create Azure OpenAI page provide the following information:
   * **Subscription**: Your Azure subscription.
   * **Resource group**: Select an existing resource group or create a new one.
   * **Region**: Choose any available region, for example, **West Europe**.
   * **Name**: Enter a unique name.
   * **Pricing tier**: **Standard (S0)**.
   ![Create an OpenAI resource.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/create-openai_hu9c72d32942d9399bae87bfb94160de88_35756_741x550_fit_q95_h2_box_3.webp)

3. Select **Review + Create** and then **Create**.
4. Once the deployment is complete, navigate to the [Azure OpenAI Studio](https://oai.azure.com/).


## Deploy a model

Before you can use the Azure OpenAI service to generate text, you need to deploy a model. There are several available [models](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models?WT.mc_id=AI-MVP-5004971) in the Azure OpenAI Studio, each of which is tailored to a specific use case. In this post, you will deploy the `text-davinci-003` model.

1. In the Azure OpenAI Studio, select **Deployments** under **Management**.
2. Click on **+ Create new deployment**, and deploy a new model with the following settings:
   * **Model name**: `text-davinci-003`
   * **Deployment name**: Choose a memorable name for your deployment.

3. Once the model is deployed, you can test it in the **Completions playground**.
   ![The Completions playground of the Azure OpenAI Studio.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/openai-playground_hu6b54be682893c3615912aa7714aa9686_48772_1098x622_fit_q95_h2_box_3.webp)

4. In the **Completions playground**, click **View code**. In the **Sample Code** window, save the endpoint of your deployed model. You will need this URL in the next step.
   ![The endpoint of a deployed model in Azure OpenAI.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/openai-endpoint_hu718247121b948338b593b880ed25474b_17320_549x201_fit_q95_h2_box_3.webp)


## Build the workflow

In a previous article, you’ve built the workflow below. This pipeline receives a form that is uploaded in an Azure Storage container, extracts the fields from the form and saves the extracted data in Azure Cosmos DB.

![Azure Logic App workflow that automates data extraction from forms using Form Recognizer.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/workflow-1_hu12eb0bdeb5033ec1849efccfcc3bdf0a_17601_501x454_fit_q95_h2_box_3.webp)

You are going to extend this pipeline by adding an HTTP request action to send a request to the Azure OpenAI service. The workflow is illustrated in the following image.

![Azure Logic App workflow that automates data extraction from forms and enrichment using Form Recognizer and Azure OpenAI.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/workflow-2_hu9456e0aa5f18221c5874bc8df7cde606_106273_1118x458_fit_q95_h2_box.webp)

You will build the form processing workflow using the Logic App Designer.

1. After the **When a resource event occurs** trigger, add two **Initialize variable** actions. Create two variables named `openai-api-key` and `openai-url` to store the Key and the URL of your Azure OpenAI resource, respectively.
   ![Initialize variables to store the key and the URL of the Azure OpenAI resource.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/set-variables_hu01b5e3ca0e73ee11353abea9b6c41fe7_18804_565x435_fit_q95_h2_box_3.webp)

      > The key of your Azure OpenAI service can be found by going to the **Keys and Endpoint** page of your resource in the Azure portal.

2. Under the **Parse Fields** block, select the plus (+) sign to add a new action. Select the **HTTP** action and enter the following information:
   * **Method**: **POST**
   * **URI**: `openai-url`
   * **Headers**: Content-Type: application/json, api-key: `openai-api-key`
   * **Body**:
       ```json
        {
          "prompt": "You must extract the following information from the review below:\n1. Sentiment (key: Sentiment) (possible values: positive, negative, neutral, mixed)\n2. Opinion for the customer service in an array (key: KeyPhrases)\nAnswer the fields briefly and provide the results in JSON format using the keys above. For the second field, summarize the opinions if needed. If the review is empty, use \"NA\" for the first field and an empty array for the second field.\nReview: \"IF EXPRESSION\"",
          "temperature": 1,
          "top_p": 0.5,
          "frequency_penalty": 0,
          "presence_penalty": 0,
          "max_tokens": 100,
          "best_of": 1,
          "stop": null
        }
       ```

      Replace `IF EXPRESSION` with the following expression, which extracts the value associated with the field `Comments`.

      ```javascript
      if(contains(body('Parse_Fields')?['Comments'], 'content'), body('Parse_Fields')?['Comments']?['content'], '')
      ```
      ![Post a request to the OpenAI API.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/openai-request_hu32e123502b158d5fdfb43b266b62490a_30742_482x484_fit_q95_h2_box_3.webp)

3. The output from the completions API will look as follows:

    ```json
    {
      "id": "ID of your call",
      "object": "text_completion",
      "created": 1680202256,
      "model": "text-davinci-003",
      "choices": [
        {
          "text": "text generated by the OpenAI API",
          "index": 0,
          "finish_reason": "stop",
          "logprobs": null
        }
      ],
      "usage": {
        "completion_tokens": 41,
        "prompt_tokens": 121,
        "total_tokens": 162
      }
    }
    ```

4. Add a **Parse JSON** action to extract the response of your OpenAI model. Specify the following details:
   * **Content**: Select **Add dynamic content** and find the block called `Body`.
   * **Schema**: To generate the schema, use a sample JSON response of the OpenAI API.
   ![Use a Parse JSON action to extract the response of the OpenAI API.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/openai-response_hud981d345ea9b3fb8429354a8c8064745_17683_592x405_fit_q95_h2_box_3.webp)

5. Then, use a second **Parse JSON** action to extract the values generated by the OpenAI API for the keys named `Sentiment` and `KeyPhrases`. Enter the following information:
   * **Content**: Select **Add dynamic content** and find the block called `text` under **Parse OpenAI response**.
   * **Schema**: To generate the schema, use a sample JSON generated by the OpenAI API or the following schema:
        ```json
        {
            "properties": {
                "KeyPhrases": {
                    "items": {
                        "type": "string"
                    },
                    "type": "array"
                },
                "Sentiment": {
                    "type": "string"
                }
            },
            "type": "object"
        }
        ```
      ![Use a Parse JSON action to extract the values generated by the OpenAI API.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/openai-values_hu51484d59e926014ed6899069a9679d19_19346_621x443_fit_q95_h2_box_3.webp)

      > The Logic App automatically adds a **For each** block around the **Parse JSON** block.

6. Modify the existing **Compose** action to include the information generated by Azure OpenAI. To extract the values associated with the fields `Sentiment` and `KeyPhrases`, select **Add dynamic content** and choose the respective blocks.
    ![Generate JSON output.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/compose_hua8985d1659ed5669887a695dcd94768b_21175_618x398_fit_q95_h2_box_3.webp)

7. Modify the **Create or update document (V3)** action as needed.
    ![Save the results in an Azure Cosmos DB database.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/save-data_hu214c80e5436c6ed96b879448ad3e8d99_24022_614x467_fit_q95_h2_box_3.webp)

Save the workflow and then click **Run Trigger > Run**. Upload a file to your Azure Storage container to test the Logic App.


## Visualize the data in Power BI

You can use Power BI to visualize the results obtained from the form processing workflow. If you don’t have a Power BI subscription, you can use Power BI Desktop, which is a free service.

1. Open Power BI Desktop and in the Home tab select **Get data > More…**
2. Choose the **Azure Cosmos DB v1** connection and click **Connect**.
3. In the pop-up window, enter the URL of your Cosmos DB account and the id of your database and collection. Then, click **OK**.
4. Once Power BI is connected to your Cosmos DB account, you can see the stored data and transform it.

Below you can see a simple Power BI dashboard that I created to visualize the data generated by the Azure OpenAI service.

![Power BI dashboard.](https://sfoteini.github.io/images/post/automate-customer-reviews-processing-with-form-recognizer-and-azure-openai/powerbi_hu7823d172265e8278b876d9617ef5aa68_60875_888x491_fit_q95_h2_box_3.webp)


## Summary and next steps

In this article, you created an end-to-end automated form processing solution using Form Recognizer, Logic Apps, and Azure OpenAI. You can use this solution to create automated workflows for your specific scenarios. You can also extend this scenario by integrating Azure Cognitive Services for Language into your Logic App workflow.

You can learn more about Azure OpenAI in the resources below:

* [What is Azure OpenAI Service? – Microsoft Docs](https://learn.microsoft.com/azure/cognitive-services/openai/overview?WT.mc_id=AI-MVP-5004971)
* [Revolutionize your Enterprise Data with ChatGPT: Next-gen Apps w/ Azure OpenAI and Cognitive Search](https://techcommunity.microsoft.com/t5/ai-applied-ai-blog/revolutionize-your-enterprise-data-with-chatgpt-next-gen-apps-w/ba-p/3762087?WT.mc_id=AI-MVP-5004971)

---

👋 **Hi, I am Foteini Savvidou!**
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 [LinkedIn](https://www.linkedin.com/in/foteini-savvidou) | [Blog](https://sfoteini.github.io/) | [GitHub](https://github.com/sfoteini)