E-Books Web Crawling Application with Recommendation System
Tags: octograd2020, githubsdp, devgrad2020
Published: 2020-05-20 15:44:45

[Comment]: # (All of this is placeholder text. Use this format or any other format of your choosing to best describe your project and experience.)

[Note]: # (If you used the GitHub Student Developer Pack on your project, add the "#githubsdp" tag above. We’d also love to know which services you used and how you used it!)

## My Final Project
Finally finished my last semester of my degree, this article is about my final year project (FYP). It is an mobile application that allows users to search for the web crawled e-books from a few websites. All the e-books are free to download. There are also recommendation systems to recommend users some e-books.


[Note]: # (Our markdown editor supports pretty embeds. If you're sharing a GitHub repo, try this syntax: `{% github link_to_your_repo %}`)

## How I built it ?
There are 4 main components for this FYP as shown below:
### Web Crawler
3 e-books websites are being scraped by using the following Python technologies:
1. Scrapy
2. Scrapyd
3. Pymongo

______________

### Mobile Application
The mobile application is being built with Flutter. Here are just a few UI designs.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/guwgxddj1o1xfqg4o0pm.png)
<center>*^this is the Library Screen^*</center>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5ob8xnmj2qr4g4bs5lwc.png)
<center>*^this is the Explore Screen^*</center>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/84errdr3dv47ysdc0ihe.png)
<center>*^this is the Story Screen^*</center>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0k9rek6e1emyj6vgk7pk.png)
<center>*^this is the Profile Screen^*</center>

_______________

### Back-end for Mobile Application
The back-end is used to handle the actions of the mobile application such as login, register, search for books and etc.
It is built with Node.js + MongoDB.

______________

### Recommendation Systems
The recommendation systems are being built with the following Python technologies:
1. flask
2. pandas
3. numpy
4. gensim
5. sklearn

There are something one might need to take care for is the cold start problem of the recommendation system. In order to solve it, the users will be prompted to select e-books and genres that they like.

There are 2 recommendation systems, the first one is a content-based recommender after users have selected their favourite e-books and genres.
The 2nd one is being used for collaborative filtering (CF) when users have rated a few e-books. The 2nd one is based on k-NN algorithm.

_____________

### Website
I've also created a [website](https://extreme-books-website.herokuapp.com/) with React to promote this application and for users to download the apk file.
Note: The website may take quite a while to load due to inactivity.
New Updates: The backend server has been shut down to save money, so the application is not working anymore.

______________

## Additional Thoughts / Feelings / Stories
In short, I am really happy that I am able to complete my FYP. Hopefully that I could get a really good grade for it.

Thank you for reading !!


[Final Note]: # (CONGRATULATIONS!!! You are amazing!)