Konohagakure Search
Tags: atlashackathon, django, mongodb, searchengine
Published: 2022-01-09 08:54:38

### Overview of My Submission
Konohagakure Search is a Google Like Search Engine. It is built on the technology of django, dedicated mongo db server, python, scrapy, spacy, and nltk. Whenever a search query is given it first search the database if its present then it searches up the internet then saves the data and then it presents to you in a pretty way :)

### Overview of the project
An efficient Search Engine with the following features:
It has distributed crawlers to crawl the private/air-gapped networks (data sources in these networks might include websites, files, databases) and works behind sections of networks secured by firewalls

It uses AI/ML/NLP/BDA for better search (queries and results) It abides by the secure coding practices (and SANS Top 25 web vulnerability mitigation techniques.) 

It is a type of a search engine which takes keyword/expression as an input and crawls the web (internal network or internet) to get all the relevant information. The application dosen't have any vulnerabilities, it complies with OWASP Top 10 Outcome. This application scrape data, match it with the query and give out relevant/related information. 

Note - Search as robust as possible (eg, it can correct misspelt query, suggest similar search terms, etc) be creative in your approach. Result obtained from search engine should displays the relevant matches as per search query/keyword along with the time taken by search engine to fetch that result.


### Submission Category: 
Choose Your Own Adventure


### Link to Code

{% github Dhruvacube/search-engine %}


### Packages that were required in making this project
- [django](https://www.djangoproject.com/)
- [dj-database-url](https://pypi.org/project/dj-database-url/)
- [celery](https://docs.celeryproject.org/)
- [django-redis](https://github.com/jazzband/django-redis)
- [django-htmlmin](https://github.com/cobrateam/django-htmlmin)
- [gunicorn](https://gunicorn.org/)
- [redis](https://github.com/redis/redis-py)
- [hiredis](https://github.com/redis/hiredis)
- [djongo](https://www.djongomapper.com/)
- [pymongo[srv]](https://pymongo.readthedocs.io/)
- [python-dotenv](https://pypi.org/project/python-dotenv/)
- [requests](https://pypi.org/project/requests/)
- [beautifulsoup4](https://pypi.org/project/beautifulsoup4/)
- [textblob](https://textblob.readthedocs.io/en/dev/)
- [spacy](https://spacy.io/)
- [nltk](https://www.nltk.org/)
- [spacy-alignments](https://pypi.org/project/spacy-alignments/)
- [spacy-legacy](https://pypi.org/project/spacy-legacy/)
- [spacy-loggers](https://pypi.org/project/spacy-loggers/)
- [colorama](https://pypi.org/project/colorama/)
- [transformers](https://github.com/huggingface/transformers)
- [Scrapy](https://scrapy.org/)
- [cdx-toolkit](https://pypi.org/project/cdx-toolkit/)
- [uvicorn](https://www.uvicorn.org/)
- [whitenoise](http://whitenoise.evans.io/)
- [colorlog](https://pypi.org/project/colorlog/)
- [uvloop](http://uvloop.readthedocs.io/)
- [spacy-transformers](https://pypi.org/project/spacy-transformers/)
- [spacy-lookups-data](https://pypi.org/project/spacy-lookups-data/)
- [django-celery-beat](https://github.com/celery/django-celery-beat)
- [django-cors-headers](https://pypi.org/project/django-cors-headers/)


### A YouTube Video Explaining all
{% youtube j1i2O2r24RM %}