Build Your Movie Recommendation System Using Amazon Personalize, MongoDB Atlas, and AWS Glue
Tags: atlasvectorsearch, generativeai, aiml, aws
Published: 2024-04-12 07:02:12

**Author:**
Siddharth Joshi, (Technical Account Manager at AWS)
Sornavel Perumal (Technical Account Manager at AWS)

**Contributor:**
Babu Srinivasan (Senior Partner Solutions Architect at MongoDB)

In today's data-driven world, personalized recommendations have become an integral part of enhancing user experiences. With the power of cloud computing and advanced database solutions, building your own personalized movie recommendation system is now more achievable than ever. In this article, we'll explore the integration of MongoDB Atlas, AWS Glue, and Amazon Personalize to create a robust and scalable recommendation engine.

## Understanding the components
 
Before diving into the integration process, let's briefly understand the key components involved in our movie recommendation system:

[MongoDB Atlas](https://www.mongodb.com/atlas/database) is a fully managed, cloud-based database service that enables seamless deployment, scaling, and maintenance of MongoDB databases.

[AWS Glue](https://aws.amazon.com/glue/) is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analysis. It helps bridge the gap between our MongoDB Atlas data and the services we'll use for recommendation.

[Amazon Personalize](https://aws.amazon.com/pm/personalize/) is a machine learning service that makes it easy to build, train, and deploy personalized recommendation models. It will analyze the data from MongoDB Atlas and generate personalized movie recommendations for users


## Reference architecture


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tbsrgvgrnmw37ir91ibt.png)

This architecture seamlessly ingests data from MongoDB Atlas, powering personalized recommendations. The AWS Glue Spark Job extracts transforms (filtering, cleaning, joining), and loads data into S3. This prepared data becomes the foundation for your chosen AI/ML service (SageMaker, Personalize, etc.), enabling highly accurate and personalized recommendations.

## Prerequisites

- [AWS account](https://aws.amazon.com/free/free-tier/)
- [MongoDB Atlas](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=bee522df-5e20-4004-9407-60ca7f22e092&sc_channel=el) free cluster
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)

This tutorial will be well understood if you have a good understanding of [MongoDB Atlas services](https://www.mongodb.com/atlas/database) and Amazon Web Services (AWS), mentioned in the above architecture diagram.

**Setting up MongoDB Atlas for movie data**

Begin by creating a MongoDB Atlas database to store information about movies, genres, and user interactions. Populate the database with relevant data, ensuring it is well-structured for the recommendation model.
For this article, we will be using the [MovieLens](https://grouplens.org/datasets/movielens/) dataset. 

a. If you do not already have one, you can [sign up](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=bee522df-5e20-4004-9407-60ca7f22e092&sc_channel=el) for a MongoDB Atlas account. 

b. Create a [database](https://www.mongodb.com/docs/atlas/atlas-ui/databases/#create-a-database) named **movielens**.  

c. Get your [connection](https://www.mongodb.com/docs/guides/atlas/connection-string/) URI to connect to MongoDB Atlas noted down.

d. [Download](https://files.grouplens.org/datasets/movielens/ml-latest-small.zip) the MovieLens dataset. 

e. Unzip the file locally and run the [Python script](https://github.com/siddharj-amz/mongo-personalize-recommender/blob/main/load_data_to_atlas.py) to upload data to MongoDB Atlas. (Replace **<connection_uri>** and **path_to_extracted_files**.)

## Using AWS Glue for data preparation
AWS Glue comes into play for ETL operations. Create a Glue job to extract data from MongoDB, transform it into a suitable format for training the recommendation model, and load it into an Amazon S3 bucket.

a. Create an [S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) to store the processed file from Glue.

b. Store your MongoDB connection properties credentials in AWS Secrets Manager.

c. Create a new AWS Glue Studio job with the Spark script editor option.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/44y8f6bzlvlsxu0p8oez.png)From the AWS Glue Studio console, select jobs from the menu and select “Script editor.” 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3rh5qd9cypi4h4eolrf.png)
Select the Spark option from the dropdown menu and click **Create script**.

d. [Create an ETL job](https://aws.amazon.com/blogs/big-data/compose-your-etl-jobs-for-mongodb-atlas-with-aws-glue/) using Glue. [Replace](https://github.com/siddharj-amz/mongo-personalize-recommender/blob/main/pyspark_atlas_to_s3.py) the Python script. 

e. Specify input arguments. 

**Key : Value**
--BUCKET_NAME : \<bucket_name\>
--OUTPUT_FILENAME1 :  ratings
--OUTPUT_FILENAME2 :  items
--COLLECTION_NAME3 :  movies
--COLLECTION_NAME2 :  tags
--COLLECTION_NAME1 :  ratings
--SECRET_NAME      :  \<name_of_secret\>

f. Run the job.


## Create a dataset group and an interactions dataset in Amazon Personalize

a. Go to Amazon Personalize in your AWS console.
b. In the left navigation pane, click on **Dataset groups**.
c. Click the **Create dataset group** button. Enter `movie-datasetgroup` as the name for your dataset group. Select **Video on demand** as the Domain. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yfkdfj4k4fz9zssxgpxc.png)
In Amazon Personalize, click **Create dataset group**, provide the name of your dataset, and select the “Video on demand” option. Click **Create group**.

d. After creating the dataset group, you need to add datasets to it. Click on the dataset group you just created.
e. Click on the **Create dataset** button, and select **Item interactions dataset**. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ly3b6xsnkc53ysiatiwh.png)
Select the “Item interactions dataset” from the dropdown menu.


f. Select Import data directly into Amazon Personalize datasets as the Import method.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l0ay32vro0zq4o60fdui.png)
Choose the “Import data directly into Amazon Personalize datasets.” 

g. Provide `movie-interactions` as Dataset name and Schema name. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sfh0sxdk96p0l43u5g9g.png)
Give a name to the Dataset and select “Create a new domain schema by modifying the existing default schema for your domain.”

h. To configure your dataset, import the job, select **Import data from S3**, and provide a name for the import job. Specify the path of ratings.csv in your S3 bucket as the data location, and specify an IAM role that has access to the S3 bucket.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5y26ttszg1kqr7lumd2y.png)
Select “Import data from S3.” 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1gbr2r803edjr0a3fyil.png)
Select the S3 data location.

## Create the User dataset

The User dataset is the dataset for all the users listed in the system. MovieLens does not provide a user dataset so we will be using one that has been created for this post. In the real world, this dataset would be coming from your application. 

a. Copy the [users.csv](https://github.com/siddharj-amz/mongo-personalize-recommender/blob/main/users.csv) file and put it on your S3 bucket created earlier. 
b. Follow the steps above to create the User dataset.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i4x75pejpu58hmunkzn4.png)
Select “Users dataset” from the dropdown menu.



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n3kmw8khh30xh6mpqidc.png)
Choose “Import data directly into Amazon Peronalize Datasets.”


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8jwrzfmi55j56rjr7ac4.png)
Provide a name for the Dataset and choose “Create a new domain schema by modifying the existing default schema for your domain.”

c. Ensure that the Schema definition looks like the following:

```
{
“type” : “record”,
“name” : “Users”,
“namespace”: “com.amazonaws.personalize.schema”,
“fields” : [
       { “name”: “USER_ID”, “type”: “string”},
       { “name”: “SUBSCRIPTION_MODEL”, “type”: “string”,”categorical”: true},
               ],
“version”:”1.0”
}
```


d. Provide S3 as the data import source and the S3 path for users.csv as the data location. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a8ql4g8ulb765dm9d502.png)
Select the “Import data from S3” option and provide the name to “Dataset import job name.”



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gazfhvoc9gbt0zros0zw.png)
Select the S3 bucket location and provide the IAM role.


## Create the Items dataset
The Items dataset refers to a list of all the movies available in our application. Our Glue ETL job has converted the MongoDB collection “movies” into a .csv file in a format usable with Amazon Personalize.
You need to follow similar steps as above to create an Items dataset. For data location, you need to provide the path of items.csv on your S3 bucket.

a. Select the “Items dataset” from the dropdown menu.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nt2wv0nw8vxeue3enw7g.png)



b. Select the “Import data directly into Amazon Personalize datasets” option.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bvhjlhuypgzkazwd32lc.png)


c. Provide a name — “movie-item” — to the Dataset name and select the “Create a new domain schema by modifying the existing default schema for your domain” option.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vxsst6rhlyrjt3px1krl.png)



d. Select the “Import data from S3” option and provide the Dataset import job name — “movie-ds-item.”
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2yuqtet3son8hvwqvc36.png)

Select the S3 location for the Data location and provide the IAM Role.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s7da22dqixydw2ru6fct.png)


e. Before proceeding to the next step, you should wait until all three datasets become active.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c6wqchy8x82vvey70dbd.png)
Ensure you can see “3/3 datasets active” in green.


## Run data analysis
Now, use Amazon Personalize for analyzing the data imported, meaning Users, Item interactions, and item datasets.
Start the data analysis by clicking on **Run data analysis**. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ra5ajitsv42m1m08b59z.png) 

Ensure the data analysis run has been completed successfully.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8xm9z1usk55zwjgaf2h5.png)
 


## Create recommenders
Create the recommenders after the Domain dataset group is created successfully. A recommender is a Domain dataset group resource that generates recommendations. Use a recommender in the application to get real-time recommendations with the [GetRecommendations](https://docs.aws.amazon.com/personalize/latest/dg/API_RS_GetRecommendations.html) operation.

a. select  **Use e-commerce recommenders**.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ot8taj16jj8ugwnfl1h4.png)


b. For the use case, select **Because you watched X** and provide a name to the recommender. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k2donq3bo217zyo6jr2k.png)


c. You can leave Advanced configuration as the default.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/luud28eicf1o0g4zz1e1.png)


d. Review the configuration and click on **Create recommenders**.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7xsrq1cvm4s0v4o4rrsv.png)


e. Before proceeding to the next step, please wait until the recommender becomes active.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6we8wob62so59yf5i9xy.png)
Ensure the status is “Active” for the movie-recommender.


## Test recommender

Now that we have created a recommender, we are ready to get recommendations. In a real-world scenario, our application would be sending requests to Amazon Personalize and getting recommendations. For the post, we will test it using the Amazon Personalize console. 
a. Go to the Amazon Personalize console.
b. Click on **Recommenders** under movie-dataset group and select **movie-recommender**.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3du1x7nwcjf3owm9toqk.png)


c. Click on **Test**. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/24b9o9nz97phoakl3y7c.png)


d. Enter a valid user ID and movie ID (Item ID), and click on **Get recommendations**.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bu19yjsm0zi54gztuxj6.png)



e. The recommender will provide the list of recommendations in the form of the movie ID. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m0c4cs0zulp3ggz8r98j.png)


In a real-world scenario, your application will map these movie IDs to movie names and will show them as recommendations to users. 

## Conclusion
In this post, we explored the integration of MongoDB, AWS Glue, and Amazon Personalize to build a personalized movie recommendation system. This powerful combination allows you to leverage the flexibility of MongoDB, the data preparation capabilities of AWS Glue, and the machine learning prowess of Amazon Personalize to deliver a tailored and engaging user experience. As you embark on your journey to enhance user engagement, this integration offers a scalable and efficient solution for building recommendation systems in various domains.


Refer to the following links for further reading:
[Writing an AWS Glue for Spark script](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-intro-tutorial.html)
[MongoDB Atlas database](https://www.mongodb.com/docs/atlas/create-database-deployment/)
[MongoDB Community Forum](https://www.mongodb.com/community/forums/)