scrapy-omdena-latam 0.0.2
Web Crawling application running Scrapy Tool extracting official policies
License: MIT
Keywords: scrapy,scraper,omdena

# Scrapy Tool for Omdena Latam LFR Challenge 
Web Crawling application running *Scrapy* Tool, extracting official policies from the following sources:

### Characteristics of the information sources

### Chile (LeyChile)

**Search type:** Exhaustive, through API , limited by the pages_num

**Speed:** Fast

**Amount of avaliable documents:** 10-100k

**Document Type:** HTML


### Mexico Distrito oficial de la Federación 
**Search type:** Exhaustive, through scrapping (Xpath) , limited by years range.

**Speed:**  Terrible slow and buggy when you change pipelines order

**Amount of avaliable documents:** 10-100k

**Document Type:** HTML

### El Peruano
**Search type:**

**Speed:**

**Amount of avaliable documents:**

**Document Type:**





# Setup Steps:
## Recommendations:
Use a virtual environment not your python system to run and also to install the dependencies.

## Install dependencies

```
pip install -r requirements.txt
```



## Scrapy settings.py
https://drive.google.com/file/d/1bjbjYSXQqZQpJdwATRCLZFSRQciiULy-/view?usp=sharing
### Warning!
S3 upload pipeline and MySQL insert pipeline doesn't work together.
Use:
```
ITEM_PIPELINES = {
    # 'scrapy.pipelines.files.FilesPipeline': 100,
    'scrapy_official_newspapers.pipelines.ScrapyOfficialNewspapersMySQLPipeline': 200,
        }
```

and 

```
ITEM_PIPELINES = {
     'scrapy.pipelines.files.FilesPipeline': 100,
    #'scrapy_official_newspapers.pipelines.ScrapyOfficialNewspapersMySQLPipeline': 200,
        }
```
The order doesn't mind.

## Database access
Setup the DB access inserting settings.json into *scrappy_official_newspapers*.

```
{
  "username": "username",
  "password": "password",
  "db_name": "db_name",
  "aws_endpoint": "your_db_instance_access"
}
```


## S3 Access
Setup scrapy *settings.py* located at *scrappy_official_newspapers*
```
AWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
FILES_STORE = 's3://wri-latin-test/'
```


## Run
from repository root:
* cd scrapy_official_newspapers
* scrapy crawl leychile
* scrapy crawl MexicoDOF

## Monitorization/Debug/Test
Through mysql table inspection, you can check how the information is being inserted.

Through https://console.aws.amazon.com/console/home after authenticating you can navigate to S3 Service, you can check the files uploaded and their properties.

## Goal
Provide a structured information system, from multiple variated sources formats containing raw official policy documents, keeping the reference to their attributes.

## Attributes
country

geo_code

level

source

title

reference

authorship

resume

publication_date

enforcement_date
url

doc_url

doc_name

doc_type

file_urls (Needed for Scrappy S3FilesPipeline, Not in DB Schema)

Not implemented like that, but the idea is to also keep track of:
* file_raw_S3_url
* file_proccesed_task_1_S3_url 

So we are able to keep track of the policies with their attributes and also have the capability of maintaining the relationships and results of different processes.

## What is this information system:
Policies Documents Stored and indexed through the integration of a relational MySQL database and AWS S3 media database service (FTP would work too), with most attributes such as the title of the act, a resume of the document, the date of publication... 


## Documentation:
https://docs.scrapy.org/en/2.2/

https://docs.scrapy.org/en/2.2/topics/media-pipeline.html#enabling-your-media-pipeline

## Recomendations:
Do not make an extensive use of the tool, probably information will be thrown because it is still in development, you can collapse the target's resources and engraving amazon's bill through this.