core-cdc 2.0.2
This library provides the mechanism to implement Change Data Capture (CDC) services...
Requires Python: >=3.9

core-cdc (CDC a.k.a Change Data Capture)
===============================================================================

It provides the core mechanism and required resources to 
implement "Change Data Capture" services...

===============================================================================


.. image:: https://img.shields.io/pypi/pyversions/core-cdc.svg
    :target: https://pypi.org/project/core-cdc/
    :alt: Python Versions

.. image:: https://img.shields.io/badge/license-MIT-blue.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/main/LICENSE
    :alt: License

.. image:: https://gitlab.com/bytecode-solutions/core/core-cdc/badges/release/pipeline.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-cdc/-/pipelines
    :alt: Pipeline Status

.. image:: https://readthedocs.org/projects/core-cdc/badge/?version=latest
    :target: https://readthedocs.org/projects/core-cdc/
    :alt: Docs Status

.. image:: https://img.shields.io/badge/security-bandit-yellow.svg
    :target: https://github.com/PyCQA/bandit
    :alt: Security

|


Installation
===============================================================================

Install from PyPI using pip:

.. code-block:: bash

    pip install core-cdc
    uv pip install core-cdc  # Or using UV...


Features
===============================================================================

**Multi-Database CDC Support**
    - MySQL Binary Log (BinLog) based change capture
    - MongoDB Change Streams for real-time event streaming
    - Extensible processor architecture for additional database engines

**Comprehensive Event Handling**
    - DML operations: INSERT, UPDATE, DELETE
    - DDL operations: CREATE, ALTER, DROP (schemas and tables)
    - Configurable event filtering by operation type

**Flexible Target Replication**
    - MySQL target for database-to-database replication
    - Snowflake target for data warehouse integration
    - Support for multiple simultaneous targets

**Standardized Data Format**
    - Common Record structure for cross-service integration
    - Includes metadata: timestamps, transaction IDs, source position
    - JSON serialization support for streaming and messaging systems

**Production-Ready Features**
    - Built-in error handling and retry mechanisms
    - Comprehensive logging for monitoring and debugging
    - Optional event timestamp column for UPSERT/MERGE operations


Quick Start
===============================================================================

Installation
-------------------------------------------------------------------------------

Install the package:

.. code-block:: bash

    pip install core-cdc
    uv pip install core-cdc     # Or using UV...
    pip install -e ".[dev]"     # For development...


Setting Up Environment
-------------------------------------------------------------------------------

1. Install required libraries:

.. code-block:: bash

    pip install --upgrade pip
    pip install virtualenv

2. Create Python virtual environment:

.. code-block:: bash

    virtualenv --python=python3.12 .venv

3. Activate the virtual environment:

.. code-block:: bash

    source .venv/bin/activate

Install packages
-------------------------------------------------------------------------------

.. code-block:: bash

    pip install .
    pip install -e ".[dev]"

Optional libraries
-------------------------------------------------------------------------------

.. code-block:: bash

    pip install '.[all]'  # For all...
    pip install '.[mysql]'
    pip install '.[mongo]'
    pip install '.[snowflake]'


Check tests and coverage
-------------------------------------------------------------------------------

.. code-block:: bash

    python manager.py run-tests
    python manager.py run-tests --test-type integration
    python manager.py run-coverage

    # Having the docker containers up and running you can execute the functional
    # tests that ensure the CDC services are working as expected...
    python manager.py run-tests --test-type functional --pattern "*.py"


Implemented CDC Engines
===============================================================================

The following database engines have CDC implementations:

Fully Implemented
-------------------------------------------------------------------------------

**MySQL** - Binary Log (BinLog) based CDC
    - Uses ``mysql-replication`` library
    - Captures INSERT, UPDATE, DELETE operations
    - Supports DDL events (CREATE, ALTER, DROP)
    - Fallback mechanism for column name resolution
    - See: `core_cdc/processors/mysql/ <core_cdc/processors/mysql/>`_

**MongoDB** - Change Streams based CDC
    - Uses native MongoDB Change Streams
    - Captures INSERT, UPDATE, DELETE operations
    - Requires replica set configuration
    - Real-time event streaming
    - See: `core_cdc/processors/mongo/ <core_cdc/processors/mongo/>`_

In Development
-------------------------------------------------------------------------------

**MS SQL Server** - Abstract base class defined
    - See: `core_cdc/processors/mssql.py <core_cdc/processors/mssql.py>`_

**Oracle** - Abstract base class defined
    - See: `core_cdc/processors/oracle.py <core_cdc/processors/oracle.py>`_


Contributing
===============================================================================

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Write tests for new functionality
4. Ensure all tests pass: ``pytest -n auto``
5. Run linting: ``pylint core_cdc``
6. Run security checks: ``bandit -r core_cdc``
7. Submit a pull request


License
===============================================================================

This project is licensed under the MIT License. See the LICENSE file for details.


Links
===============================================================================

* **Documentation:** https://core-cdc.readthedocs.io/en/latest/
* **Repository:** https://gitlab.com/bytecode-solutions/core/core-cdc
* **Issues:** https://gitlab.com/bytecode-solutions/core/core-cdc/-/issues
* **Changelog:** https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/master/CHANGELOG.md
* **PyPI:** https://pypi.org/project/core-cdc/


Support
===============================================================================

For questions or support, please open an issue on GitLab or contact the maintainers.


Authors
===============================================================================

* **Alejandro Cora González** - *Initial work* - alek.cora.glez@gmail.com