AWS re:Invent 2022 - Classify German legal documents with Amazon Comprehend IDP - Part 1
Tags: aws, machinelearning, nlp, tutorial
Published: 2022-12-22 10:45:21

## AWS re:Invent 2022 | AI and Machine Learning

One of the [new announcements from AWS re:Invent 2022](https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-comprehend-support-classification-entity-exctraction-document-formats/) in the AI and Machine Learning category was Amazon Comprehend Intelligent Document Processing (IDP) which supports other document file formats such as PDF, Microsoft Word documents 
and images.

### Background

The existing Amazon Comprehend natural language processing service can mine text files to perform analyses such as:

* Key Phrase
* Topic Modelling
* PII detection
* Sentiment Analysis
* Targeted Sentiment Analysis
* Custom Classification
* Custom Entity Recognition

My solution architecture for ingesting a single text file is provided below:

![solution](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ezh5wq382wg4iqjbaz7q.jpg)

What are the business use cases for Amazon Comprehend?

* Customer Support Tickets
* Mine call centre analytics
* Extract customer sentiment, key phrases from customer surveys
* Analyze customer interactions
* Find key topics from customer feedback
* Classify and extract entities from documents

### Learning Objectives

In this lesson, you will learn:

*  How to classify and extract entities with the new feature from Amazon Comprehend IDP

### Amazon Comprehend Intelligent Document Processing (IDP) 

Amazon Comprehend Intelligent Document Processing (IDP) has greater flexibility to directly process text and extract custom entities from PDF, Microsoft Word documents and images.

Custom entity recognizers for PDF documents can only be used for English language as at [21 December 2022](https://docs.aws.amazon.com/comprehend/latest/dg/custom-entity-recognition.html).

This is the reference architecture provided by Amazon Web Services:

![reference](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/68tm2dxfc6ecbs95yzve.jpg)

At AWS re:invent 2022, PDF files can be used to extract insights from entities and classify documents.

![idpn](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/la022ellaf0exhd1mz8b.jpg)

### What are the business use cases for Amazon Comprehend IDP?

The [use cases](https://aws.amazon.com/comprehend/idp/) include:

* Extract information from insurance claim forms
*  Classify and extract entities from income statements to complete loan applications
* Real-time processing of documents (i.e. synchronous)
* Batch-processing of large documents (i.e. asynchronous)
* Classify and extract insights from legal documents
* Extract information from tax invoices

### Dataset

This dataset contains German legal documents in PDF format that was downloaded from the website [Papers with Code](https://paperswithcode.com/dataset/dataset-of-legal-documents).

The legal documents consists of court decisions from 2017 and 2018 were selected for the dataset, published online by the Federal Ministry of Justice and Consumer Protection. 

#### Citation:

@misc{https://doi.org/10.48550/arxiv.2003.13016,
  doi = {10.48550/ARXIV.2003.13016},
  url = {https://arxiv.org/abs/2003.13016},  
  author = {Leitner, Elena and Rehm, Georg and Moreno-Schneider, Julián},  
  keywords = {Computation and Language (cs.CL), Information Retrieval (cs.IR), FOS: Computer and information sciences, FOS: Computer and information sciences},  
  title = {A Dataset of German Legal Documents for Named Entity Recognition},  
  publisher = {arXiv},  
  year = {2020},  
  copyright = {arXiv.org perpetual, non-exclusive license}
}

### Two Approaches - Annotation of PDFs or Entity Lists

There are two methods to extract entities from [PDF files](https://docs.aws.amazon.com/comprehend/latest/dg/prep-training-data-cer.html):

* Method 1: An advanced approach involves a few pre-requisites to set up the environment and label documents with Amazon Ground Truth and

* Method 2: A simpler approach includes preparing training data via a plain text files in a csv file UTF-8 format with two headings 'Text' and 'Type'. With a minimum of 25 entities (i.e. Type) listed in the entity list


### Tutorial 1: Method 1: Custom Entity Recognition with Amazon Comprehend Intelligent Document Processing (IDP) for [German legal documents translated into English](https://huggingface.co/datasets/elenanereiss/german-ler#citation-information) - with Entity Lists.

* Step 1: Sign into the AWS Management Console with IAM Administrator role.

Navigate to **Amazon Comprehend** and click **Launch Amazon Comprehend**.


![navigate](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g617kerr1e938e21cpqb.jpg)

* Step 2: Select **Analysis jobs** from the left hand-side and **Create job**.


![analysis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2qcbg62msn6700gru23j.jpg)

* Step 3: Create analysis

Provide a name for the analysis, select from the drop-down menu
'Entities' as the analysis type.

Select 'English' as the language.


![entity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8zo04esnb33w9i26ccnf.jpg)

For the input document, select the training dataset from the S3 bucket and the file type as 'document per line' because it is a small file size.


![Create iam](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u21qzpydqw04lnvzdror.jpg)

Create an IAM role for Amazon Comprehend role and click **Create job**.


* Step 4: The analysis job will take a few minutes to complete processing.


![job2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dabbhdn6gauitms1q09g.jpg)

Let's inspect the output of the analysis job:

Download and open the output file in gz format using 7-Zip. If you do not have 7-Zip you may access it [here](https://www.7-zip.org/download.html)

* Step 5:  Prepare training data [entity list](https://docs.aws.amazon.com/comprehend/latest/dg/cer-entity-list.html) for building a custom recognition model

The csv file for preparing training data is in UTF-8 format with headers 'Text' and 'Type'. The 'Type' is in uppercase as per AWS documentation in preparing data.

![text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fkipsmfrzlfc0eri1j8h.jpg)

* Step 6: Create a Custom Entity Recognition model.

In Amazon Comprehend click **Create new model**

![Create new model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zbagsspykawgyqe8kyca.jpg)

Add custom entity types.


![custom](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4s2jlp5uvd1xzdqpwqq6.jpg)

Upload 'entity list' as a csv file into Amazon S3 bucket as training data.


![up into s3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/17dx0fh4adgktoaemycz.jpg)

Upload 'german_test' csv file into the Amazon S3 bucket for the test data.


![upload test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tcsfdrezpsfosjkfd9ue.jpg)

Navigate and select the second option for 'custom entity list' as a source of training data, select output location for trained data and select the custom test data that is stored in the Amazon S3 bucket.


![model train](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t14chouj0znisgkw6mkq.jpg)

Select the existing IAM role created for Amazon Comprehend and click **Create**.


![training modeln](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bnik68g7gkh8h5pw4j4n.jpg)

Model is submitted for processing.


![modeln](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ozpdpounfxg5xjm3etw0.jpg)

Tip 1: In the training data, remove any special characters e.g '/'.

Tip 2: Each custom entity must have a minimum occurrence of 25 times to be included in the plain text file of an entity list.


![entity appear](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/65o3es4041ztuytl4cx5.jpg)


I updated the fourth model for training purposes.



![4th model](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ke7zawclept73mwsy9ms.jpg)

After 15 minutes, the classifier model is trained.


![trained](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m1eeh6d548fna1zbr4op.jpg)

The model produced a F1 score of 77.30 on the training data set.

![F1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cppxeye1y1pv48i7s8yo.jpg)

Step 7: Create a custom entity detection analysis job (synchronous processing)

Finally, create an **analysis job** selecting 'custom entity recognition' as the analysis type. Select the test data from S3 bucket.


Select the output data location in the S3 bucket, select the IAM role and click **Create job**.



The custom entity analysis will be processed and take a few minutes to complete.


![model 2n](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h03fy0u7bq9wbxgk8ncr.jpg)


Let's inspect the output of the custom entity recognizer analysis for legal documents stored in the S3 bucket.


![output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19yc26fm4k0p5bl4r68t.jpg)

### Results of Custom Entity Recognition model from Entity Lists

The output of Custom Entity Recognition model using test data on the trained classifier model produced the following results.

All the entity types extracted from the test data file recognized the input text with a 99% confidence level. 

![testing data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wsiapmiivd7su7hffib3.jpg)

The exceptions were the last seven input text which had a lower confidence level in recognizing the entities for the German organizations from the legal documents.



![last seven](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mxju2r5fpndkviiry014.jpg)


### Method 2: [Pre-requisites](https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html) - Checklist before Annotating PDF Files


#### Pre-requisite 1: Create a virtual environment to use the latest version of Python in AWS Cloud9

* Step 1:  Create an AWS account for IAM Administrator - refer to my [blog](https://dev.to/aws-builders/creating-your-first-iam-admin-user-and-user-group-in-your-aws-account-machine-learning-part-1-3cne)

Login to the AWS Management Console account as IAM Administrator.

* Step 2: Create a virtual environment to use the latest version of Python in AWS Cloud9.

Navigate and type in the search bar Cloud 9.


![cloud 9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d3buzkatg6020efxahep.jpg)

Select **Create environment**. 




![env](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vbkzyml6xqbmmphshhu0.jpg)

Create a name for the temporary environment and select the smallest EC2 instance i.e. t2.micro and then click **Create**.


![micron](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/stjkbgwljorilpldr0ga.jpg)

* Step 3: Configure the virtual environment with your AWS credentials.


Open the Cloud 9 IDE and type configure your AWS virtual environment.


![open IDE](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5dm2r911oriu6avai11b.jpg)

Type 'aws --version' to check the version of the AWS CLI.


![clion](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rgf3uuet19cw2c88585x.jpg)

And refer to this [link](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) to configure the AWS environment

#### Pre-requisite 2: Setting up the [environment](https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html#cer-annotation-pdf-set-up)

* Step 1: Install cygwin for Windows by clicking this [link](https://cygwin.com/install.html)


![cygwinn](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/28c7hof1z99fkj9mfiua.jpg)

* Step 2: Download the annotation files from [Github](https://github.com/aws-samples/amazon-comprehend-semi-structured-documents-annotation-tools)


* Step 3: Create a virtual environment in Python

In the Terminal of Visual Studio, type 'Python':

$ pip install virtualenv

![python virtual](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hr9jcwt34vv1xbv0ylo0.jpg)

$ pip install --upgrade pip          

![pip py](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rko58haif9j5twpi665r.jpg)

* Step 4: Unzip the Github files

![githubon](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4i8sy9yvcsob6pmgw3km.jpg)

Unzip the annotation files folder downloaded from Github in your IDE.

I used Visual Studio IDE by going to **Extensions** on the left hand-side menu and installing 'AWS Toolkit'

Go to **View** -> **Colour Palette** -> **AWS Create Credentials**:

Provide your profile details from IAM Administrator:

AWS Access Key ID:
AWS Secret Access Key:


![unzip](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e9zda6zyibn5kfjg41en.jpg)



#### Pre-requisite 3: Create an Amazon S3 bucket (using Amazon Management Console) -refer to my [blog](https://dev.to/aws-builders/creating-your-first-amazon-s3-bucket-and-new-feature-release-2c03).

* Step 1: Create a S3 Bucket called 'src'.


![src](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h41r36clgefnb23a9zw9.jpg)

Retain the default settings and select **Create bucket**.


* Step 2: Upload the train dataset in PDF format into the S3 bucket you have created.


![upload](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z2dfg3br057usxinxm65.jpg)


#### Pre-requisite 4: Annotate PDF files

In the next lesson, we will annotate PDF files from the German legal document and complete the pre-requisites from [here](https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html).


### Resources

* [Amazon Comprehend Intelligent Document Processing (IDP)](https://aws.amazon.com/comprehend/idp/)

* [Analyzing Sentiment from text](https://aws.amazon.com/getting-started/hands-on/analyze-sentiment-comprehend/)

* [AWS re:Invent 2022 announcement](https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-comprehend-support-classification-entity-exctraction-document-formats/)

* [Limits and Quotas](https://docs.aws.amazon.com/comprehend/latest/dg/guidelines-and-limits.html)

* [Preparing training data](https://docs.aws.amazon.com/comprehend/latest/dg/prep-training-data-cer.html)

* [Annotation best practice](https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation.html)

* [Create a custom entity recognition detection job](https://docs.aws.amazon.com/comprehend/latest/dg/detecting-cer-async-console.html)

### AWS re:Invent 2022 keynotes, workshops and leadership sessions

In case you missed AWS re:Invent 2022 a few weeks ago you can experience the excitement by learning about the latest innovation and product features [here](https://www.youtube.com/@AWSEventsChannel).


![reinvent](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2s4bnbgzrkid7q1oxhzy.jpg)

Until the next lesson, happy learning! 😁

### Next Lesson: AWS re:Invent 2022 - Document AI: Classify German legal documents with Amazon Comprehend IDP - Part 2

Custom Entity Recognition Analysis with Amazon Comprehend Intelligent Document Processing (IDP) for [German legal documents translated into English](https://huggingface.co/datasets/elenanereiss/german-ler#citation-information) - for Annotated PDF files with labeling using Amazon Ground Truth.