mnqueues 0.0.39
Monitored Multiprocessing Queues in Python
License: MIT
Requires Python: >=3.9

[![CircleCI](https://circleci.com/gh/amor71/mnqueues/tree/main.svg?style=shield)](https://circleci.com/gh/amor71/mnqueues/tree/main)
[![Updates](https://pyup.io/repos/github/amor71/mnqueues/shield.svg)](https://pyup.io/repos/github/amor71/mnqueues/)
[![Python 3](https://pyup.io/repos/github/amor71/mnqueues/python-3-shield.svg)](https://pyup.io/repos/github/amor71/mnqueues/)
[![Sourcery](https://img.shields.io/badge/Sourcery-enabled-brightgreen)](https://sourcery.ai)

# What are mnqueues?

`mnqueues` stands for Monitored Queues - a coupling between Python `multiprocessing` Queue() and a Monitor entity. A Monitor collect and alerts on Queue usage statistics. 
## Tracked measures

`mnqueues` tracks several measures per queue:

1. Average number of writes to queue per minute,
2. Average number of reads to queue per minute,
3. Time spend in queue (latency) in milliseconds. 
## Installation

To install `mnqueues` type: 

`pip install mnqueues`

## How-To use

To create a monitored queue:

    import mnqueues as mnq
    from mnqueues.gcp_monitor import GCPMonitor

    q = mnq.MNQueue(monitor=GCPMonitor("some-unique-name"))

The `MNQueue()` object encapsulated Python `multiprocessing.Queue()` and supports same functions. The MNQueue() object can be passed between processes, like a Queue() object. 

## Monitors

### File Logger

    from mnqueues.log_monitor import LOGMonitor

    monitor = LOGMonitor("log-file-name")

Log all `put()` and `get()` calls to a log file with the the name `log-file-name.log` with the following format:

    [<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 5003
    [<OS process-id>]->2021-07-07 21:31:14 INFO:get counter: 4997


### Google Cloud Monitor (using StackDriver)

    from mnqueues.gcp_monitor import GCPMonitor

    monitor = GCPMonitor("unique-name")

All calls to `put()` and `get()` are sent to Google Cloud Monitor. The Monitor class sends data to two custom measures:
1. `OpenCensus/mnqueues.{name}.number_queue_get` (line, no aggregation on GCP required)
2. `OpenCensus/mnqueues.{name}.number_queue_put` (line, no aggregation on GCP required)
3. `OpenCensus/mnqueues.{name}.time_in_queue_distribution` (heat-map with sum, shows latency distribution)

Note that `{name}` is passed as a parameter when constructing the Monitor and it aims to assist in creating dash-boards for specific use-cases. 

[See](https://amor71.github.io/mnqueues/gcp-metric-explorer.png) for details.

## Examples

### GCP
1. [View](https://amor71.github.io/mnqueues/gcp-mnqueues-sample.png) Google Cloud (GCP) Monitoring dashboard showing queue.put() and queue.get() rates per second, generated by running `pytest` on the project [tests](https://github.com/amor71/mnqueues/tree/main/tests) folder.
2. [Monitoring](https://amor71.github.io/mnqueues/liu-monitor.png) queues with real-time web-socket trading data for [LiuAlgoTrader](https://github.com/amor71/LiuAlgoTrader).

### Further examples

Can be found in the [tests](https://github.com/amor71/mnqueues/tree/main/tests) folder.

## Contributing

Contributions are highly appreciated. Please review our 
[Code of Conduct](https://github.com/amor71/mnqueues/blob/master/CODE_OF_CONDUCT.md). Bug reports & feature requests can be left in the `Issues` section, or email me at amor71@sgeltd.com