semvec/pypi-packages huggingface dataset:
pg-replicate-elastic 1.0.3
Replication PostreSQL to Elasticsearch via Logical replication slots
License: MIT
Keywords: postgresql,elasticsearch,replication

## Replication PostreSQL to Elasticsearch via Logical replication slots


### Install

```
$ pip install pg-replicate-elastic
```

#### wal2json

*Note:* Use my fork as we tested it to be 100% sure it work with it and no breaking changes happen, [wal2json](https://github.com/hmilkovi/wal2json) .

```
$ git clone https://github.com/hmilkovi/wal2json.git
$ PATH=/path/to/bin/pg_config:$PATH
$ USE_PGXS=1 make
$ USE_PGXS=1 make install
```

You need to set up at least two parameters at postgresql.conf:
```
wal_level = logical
max_replication_slots = 1
```
After changing these parameters, a restart is needed.

### Usage

```
pg_replicate_elastic --config=<absolute path to json config>
```

Construct configuration file in json format where:

* **replication_slot** json object for replication slot name and if is temporary
* **tables** is array of tables we want to replicate
* **es_connection** connection string to Elasticsearch
* **postgres** json object for PostreSQL connection
* **inital_sync** boolean for inital syncronization that needs to be done first time
to replicate old data

Example configuration
```
{
	"replication_slot": {
		"name": "elasticsearch_slot",
		"is_temp": true
	},
	"tables": [{
		"name": "poc",
		"primary_key": "a"
		"exclude_columns": "c,z"
	}],
	"es_connection": "http://127.0.0.1:9200/",
	"postgres": {
		"port": 5432,
		"host": "127.0.0.1",
		"database": "poc",
		"username": "test",
		"password": "test"
	},
	"inital_sync": false
}
```

### Note

wal2json is not my software so for licence check [their licence](https://raw.githubusercontent.com/hmilkovi/wal2json/master/LICENSE)
django-cdc 2.0.11
CDC for django models
License: MIT

============================
django-cdc
============================

Tracking changes in django models and push it to logger/aws endpoints.


Quickstart Guide
===============================

Install it with pip from PyPi::

    pip install django-cdc

before installing djangoCDC, make sure that you have already run the following command or
your environment the following packages:

   pip install psycopg2
   pip install django-bitfield
   pip install boto3
   pip install pyyaml
   pip install django-extensions
   pip install confluent-kafka

Add ``django_cdc`` to your ``INSTALLED_APPS``::
   INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_extensions',
    'django_cdc',
)

If you want to track full model changes, then you need to attach an ``DjangoCDC`` manager to the model.
For foreign key attributes, we need to pass kwargs foreign of dict type and partition_key is only used with kafka::

    from django.db import models
    from django_cdc.models.managers import DjangoCDC
    from django_cdc.models.constants import ServiceType


    class ProductCategory(models.Model):
        name = models.CharField(max_length=150, primary_key = True)
        description = models.TextField()
        django_cdc = DjangoCDC(services=[ServiceType.SNS], partition_key=name,
                        service_custom_name= {ServiceType.ASYNC_KAFKA_PRODUCER : ''})

    class Product(models.Model):
        name = models.CharField(max_length = 150)
        description = models.TextField()
        price = models.DecimalField(max_digits = 10, decimal_places = 2)
        productcategory = models.ForeignKey(ProductCategory)
        django_cdc = DjangoCDC(foreign_keys={'productcategory':['name','description']},services=[ServiceType.SNS,ServiceType.KINESIS],
                        service_custom_name= {ServiceType.SNS: '' , ServiceType.KINESIS: ''})

ServiceType is used for specifying on which you want to publish and service_custom_name is used
to specify own custom name for resource- KAFKA, SNS, KINESIS. Let us say,if you want to publish
data to kafka topic like ProductCategory in that case you need to pass service_custom_name as an
argument with the custom name as shown above. Also if "None" value is set for the
service_custom_name then it will publish to the topic derived e.g. <SERVICE_FUNCTION_PREFIX>-<tableName>.

Run the following commmand to deploy lambda function that pushes data to kinesis:
        python manage.py setservice

By default warn, error, critical logging level are enabled.

To enable info logging:
        python manage.py setservice --logging_level=info

To enable debugging:
        python manage.py setservice --logging_level=debug

Following fields handled for json serialization:
        BitHandler
        datetime
        time
        ImageFieldFile