Building a Recommender System with Flask and AWS Personalize (Part 2): Setting Up Amazon Personalize
Tags: flask, restapi, aws, tutorial
Published: 2022-05-11 15:11:23

In the previous part of this series, we learnt what a recommender system is. We also learnt about some use cases of a recommender system, and how Amazon Personalize works. Here, we would be going through the steps required to set up Amazon Personalize.

## Setting Up Amazon Personalize
To get started with Amazon Personalize, you need to have an Amazon Web Service (AWS) account. In case, you don't have an account, follow the instructions [here](https://docs.aws.amazon.com/personalize/latest/dg/setup.html#aws-personalize-set-up-aws-account) to create an account. 

Now that you've created your AWS account, you need to create an AWS Identity and Access Management (IAM) admin user. An IAM user is an entity created to represent a person or application interacting with an AWS. An IAM user with administrative permission has unrestricted access to AWS services in your account. To create an IAM user, follow the set of instructions [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html). 

Before we can use Amazon Personalize, you have to set up permissions that allow your IAM user access to Amazon Personalize Console and API operations. To set up the IAM user policy, follow the instructions [here](https://docs.aws.amazon.com/personalize/latest/dg/aws-personalize-set-up-permissions.html#set-up-required-permissions).

You now have to create an IAM role for Amazon Personalize. An IAM role is an IAM identity that you create in your account that has specific permissions. An IAM role is similar to an IAM user, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, an IAM role is not uniquely associated with one person, a role is intended to be assumable by anyone who needs it. To create the IAM Role, follow the instructions [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html).
Use these configurations while creating the IAM Role:
* For **Choose the service that will use this role**, choose `Personalize`.
* For **Attach permissions policies**, either choose the IAM user policy you created previously, or choose `AmazonPersonalizeFullAccess` from the list of Amazon-managed policies.

After creating the IAM role for Amazon Personalize, you have to prepare your training data and upload it to Amazon S3. We would be using a Custom dataset group. Download the user-interactions file [here](https://drive.google.com/file/d/1xq78NlKjkhnPafeeLjoAseG0JUJBAW87/view?usp=sharing). 
You would have to format the `user_interaction.csv` so that Amazon Personalize can understand the data. We have to change the header rows from `user_id`, `movie_id`and `timestamp` to `USER_ID`, `ITEM_ID`, and `TIMESTAMP`. You can manually enter it with an Excel file or use the Python Pandas package. 
Follow these steps to change the header row using pandas
```
import pandas as pd
# Read user_interactions.csv into a Pandas Dataframe
df = pd.read_csv("user_interactions.csv", encoding='latin-1')
#Rename column name
df = df.rename(columns={"user_Id":"USER_ID","movie_Id": "ITEM_ID","timestamp": "TIMESTAMP"})
#Save dataframe into a new CSV file named interactions.csv
df.to_csv("interactions.csv", index=False)
```
We would have to upload our `interaction.csv` file into an AWS S3 bucket. To create an S3 bucket and upload files to it follow the set of instructions [here](https://docs.aws.amazon.com/redshift/latest/dg/tutorial-loading-data-upload-files.html). 
For Region, choose `US West (Oregon) us-west-2`.

After uploading your file to Amazon S3, you have to set up permission that allows Amazon Personalize to access the Amazon S3 bucket. We would be doing two things to set up the permission:
- Attaching an Amazon S3 policy to your Amazon Personalize service role.
- Attaching an Amazon Personalize access policy to your Amazon S3 bucket. 

To attach an Amazon S3 policy to your Amazon Personalize service role, do the following:
- Sign in to the [IAM console](https://console.aws.amazon.com/iam/).
- In the navigation pane, choose Policies, and choose **Create Policy**.
- Choose the JSON tab, and update the policy as follows. Replace the bucket name with the name of your S3 bucket.
   ```
{
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}
   ```
- Choose Next: Tags. Optionally add any tags and choose Review.
- Give the policy a name.
- For **Description**, enter a short sentence describing this policy, for example, `Allow Amazon Personalize to access its Amazon S3 bucket`.
- Choose the **Create Policy**.
- In the navigation pane, choose Roles, and choose the role you created for Amazon Personalize.
- For Permissions, choose **Attach Policies**.
- To display the policy in the list, type part of the policy name in the **Filter policies** filter box.
- Choose the check box next to the policy you created earlier in this procedure.
- Choose **Attach policy**.

Now you have to attach a bucket policy to the Amazon S3 bucket containing your data. This permission allows Amazon Personalize access to the S3 bucket. 

To create a bucket policy:
- In the **Buckets** list, choose the name of the bucket that you want to create a bucket policy.
- Choose **Permissions**.
- Under **Bucket Policy**, choose **Edit**. This opens the Edit bucket policy page.
- On the **Edit bucket policy** page, replace the JSON in the **Policy** section, with the following:
```
{
    "Version": "2012-10-17",
    "Id": "PersonalizeS3BucketAccessPolicy",
    "Statement": [
        {
            "Sid": "PersonalizeS3BucketAccessPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "personalize.amazonaws.com"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}
``` 
Replace `bucket-name` with the name of the S3 bucket.

- Choose the **Save Changes** to save the policy.

Before starting with Amazon Personalize, you have to determine your use case. Determining your use case will help identify what recipe would be used to train our model. **Recipes** are Amazon Personalize algorithms that are prepared for specific use cases. 
The available use cases for Amazon Personalize include:
- Recommending items for users using the `USER_PERSONALIZATION` recipe.
- Ranking items for a user using the `PERSONALIZED_RANKING` recipe.
- Recommending similar items using the `RELATED_ITEMS` recipe.
- Getting user segments using the `USER_SEGMENTATION` recipes.

Our use cases are for recommending items to users and recommending similar items. To start with Amazon Personalize, we have to create a Custom Dataset group, to do this follow the set of instructions [here](https://docs.aws.amazon.com/personalize/latest/dg/data-prep-ds-group.html#data-prep-creating-ds-group-console). 
After creating a new Dataset group, you would be directed to the **Create interactions dataset** page. Follow the following steps to create an interaction dataset and schema:
- Specify a name for the dataset group.
- Choose **Create new schema**.
- Enter a name to distinguish our schema.
- Paste the schema JSON in the **Schema definition**.
```
{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    {
      "name": "USER_ID",
      "type": "string"
    },
    {
      "name": "ITEM_ID",
      "type": "string"
    },
    {
      "name": "TIMESTAMP",
      "type": "long"
    }
  ],
  "version": "1.0"
}
```
- Choose **Create dataset and continue**

Next, we have to import our interaction data into Amazon Personalize from Amazon S3. To do these, we would have to create an import job. Follow the steps [here](https://docs.aws.amazon.com/personalize/latest/dg/bulk-data-import-step.html#bulk-data-import-console) to create a data import job.
After creating the dataset import job, you will be redirected to the **Dashboard Overview** page. 

On the create dataset row, you would see the Interaction data would have Pending status. Amazon Personalize would take about   5-15 mins to create the Interaction dataset.
While we wait for Amazon Personalize to create the Interaction dataset, we would import the Item dataset. Before we import our Item dataset into Personalize, we need to have our Item metadata. You can download the Item metadata file we would be using in this tutorial [here](https://drive.google.com/file/d/18Pl7xzIp4_NkMOG27yyJURRzCEb_vr_j/view?usp=sharing). Click [here](https://docs.aws.amazon.com/personalize/latest/dg/item-dataset-requirements.html#schema-examples-items), to learn more about creating an Item dataset. 
After downloading our Item metadata we have to upload the file to our previously Amazon S3 bucket. To upload the Item metadata to S3:
1. Open the Amazon S3 console.
2. In the **Buckets**  list, choose the bucket's name to which you want to upload your files to.
3. Choose **Upload**.
4. In the Upload window, drag and drop the item metadata file to the Upload window.
5. Choose **Upload** at the bottom of the page to upload the file to our Amazon S3 bucket.

To import our Item dataset into Amazon Personalize;
1. Select the **Import Item data ** on the **Overview Dashboard**.
2. Enter a dataset name of your choice that will be easily distinguishable.
3. Select **Create new Schema**.
4. Enter the following JSON schema, in the Schema Definition.
```
{
	"type": "record",
	"name": "Items",
	"namespace": "com.amazonaws.personalize.schema",
	"fitelds": [
		{
			"name": "ITEM_ID",
			"type": "string"
		},
		{
			"name": "YEAR_RELEASED",
			"type": "int"
		},
		{
			"name": "GENRE",
			"type": "string",
			"categorical": true
		},
		{
			"name": "ACTORS",
			"type": "string",
			"categorical": true
		},
		{
			"name": "DIRECTOR",
			"type": "string"
		}
	],
	"version": "1.0"
}
```
Choose **Next**, you would be directed to the **Configure dataset import job** page.
Enter the dataset import job name the **Configure dataset import job** page. The dataset import job name must be between 1-63 characters and shouldn't contain space.

Enter the S3 location of your data. If you don't know the location of the file in the S3 bucket, go to the file location in the S3 console. Check the box of the file and choose the **Copy S3 URI**. 
Paste the S3 URI into the S3 location box on the **Configure Dataset import job** page.

In the **IAM Role** section, choose **Enter a custom IAM Role ARN**. You have to enter the ARN of the IAM Role previously created. To get the ARN, go to the IAM console. Choose **Role** and choose the IAM role created earlier. On the IAM Role page, you would see ARN. Copy the ARN and paste it into the IAM Role ARN on the **Configure dataset import job**. Choose **Start Import**.

After importing data, we have to create a Solution. A Solution is a combination of Amazon Personalize recipe, customized parameters, and one or more solution versions (trained models) that Amazon Personalize uses to generate recommendations. Amazon Personalize provides recipes based on common use cases, for training models. Recipes are Amazon Personalize algorithms that are prepared for specific use cases. You can read more about Amazon Personalize recipes, [here](https://docs.aws.amazon.com/personalize/latest/dg/working-with-predefined-recipes.html)

To create the Solution we would be using, head over to the Amazon Personalize **Overview** page, and choose **Create Solution**.

On the **Create Solution** page. Enter a name for the solution. The solution name must have 1-63 characters with no spaces. (Valid characters: a-z, A-Z, 0-9, and _). 

For Solution type, choose **Item recommendation**.

For **Recipe**,  select  `aws-similar-items`. The `aws-similar-items` or Similar-Items recipe generates recommendations for items that are similar to an item you specify.

Select `Create and train solutions`. You would then be redirected to the **Overview** page.

After preparing and importing our data and creating our solutions, we are ready to deploy our solution version to generate recommendations in real-time. A solution version is a trained model that is created as part of a solution in Amazon Personalize.  We can deploy our solution version by creating an Amazon Personalize campaign. A campaign is a deployed solution version (trained model) with provisioned dedicated transaction capacity for creating real-time recommendations for your application users. After you create a campaign, you use the `GetRecommendations` or `GetPersonalizedRanking` API operations to get recommendations.

Follow the instructions, [here](https://docs.aws.amazon.com/personalize/latest/dg/campaigns.html#create-campaign-console) to create a campaign. This campaign will be used to get recommendations of items similar to a selected item.
In the next part of this series, we would creating a movie API and integrating Amazon Personalize into our API.