gatherer 0.0.1
Collects data from multiple MySQL Databases in a multi-tenant architecture
Keywords: python,mysql,database,collector,gatherer,csv,multi-tenant


# Gatherer
Collects data from multiple MySQL Databases in a multi-tenant architecture. Gatherer helps to gather the scattered data across different databases which share the same schema. It outputs the whole collected data into a single CSV file.

##### Pip
You can install this package from pip by running the following command :
```html
pip install gatherer
```

#### Pros

- Cuts out the need of performing a manual job by executing your query on each database and then merging them.
- No wait for huge response which may fail or exceed different limits.
- Specific databases may be excluded.
- Huge and complex queries can be read from a file.
- Catalog can be loaded from three different sources (manual list, the same instance as tenant, the remote instance).
- Eliminates the need of advanced knowledge in cross-database queries.
- The process can be terminated at any time, and it can be restarted later.

#### Parameters

Name | Description | Type | Default
--- | --- | --- | --- 
host | The MySQL Instance IP/Identifier of Tenant Database/s (or all-with Catalog). | string | localhost
user | The MySQL Instance User of Tenant Databases (or all-with-Catalog). | string | root
password | The MySQL Instance Password of Tenant Databases (or all-with-Catalog). | string | Empty
port | The MySQL Instance Port of Tenant Databases (or all-with-Catalog). | integer | 3306
output_dir |  Each tenant file will be stored here before the final merge. Consider it as a temporary database. | string | output_dir
tenant_query | The SELECT query to be performed. | string | None
tenant_query_source | The SELECT query to be performed from file. | string | None
tenant_dbs | Databases given manually. If this is set then Catalog parameters will not be taken into consideration. | array | None
skip_tenant_dbs | Tenant Databases to be skipped. | string | None
catalog_db | The Catalog Database. The Catalog Query will be executed here. | string | None
catalog_query | The Query to SELECT Tenants Database names. | string | None
catalog_host | The MySQL Instance IP/Identifier of Catalog.  | string | None
catalog_user | Catalog host User. | string | None
catalog_password | Catalog host Password | string | None
output_file_name | The name of the file which contains the final result. | string | output
skip_processed_tenants | Tenant Databases to be skipped. | string | None
file_encoding | The File Encoding to be used in final CSV. | string | latin-1
file_headers | The Headers in final CSV file. | string | latin-1
debug | Debug the process on console. | boolean | False



### How it works
Collector runs a given query in each tenant database which is present in its {list}, this list is either given manually (filled with Database names) or fetched automatically by a given Catalog query.
Each of these runs yields a result which is outputted into a CSV file in {output_dir}. At the end of this, all files are merged into a single one.

### Simple usage

```python

import gatherer

gatherer.gather(host='localhost',
                  user='root',
                  password='',
                  port=3306,
                  tenant_query='SELECT my_column_1,my_column_2,my_column_3 FROM my_table;',
                  skip_tenant_dbs=['My_Tenant1_DB', 'My_Tenant2_DB'],
                  catalog_db='CATALOG_DB',
                  catalog_query='SELECT my_tenant_database_name FROM my_tenants;',
                  output_dir='temp_output'
                  )

```