pymemtrace 0.2.0
Python memory tracing.
License: MIT license
Keywords: pymemtrace

*******************
Introduction
*******************


``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.

Full documentation: https://pymemtrace.readthedocs.io

pymemtrace Tools
======================

The tools provided by ``pymemtrace``:

* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.
  It can plot memory over time with plotting programs such as ``gnuplot``.
  See `some process examples <https://pymemtrace.readthedocs.io/en/latest/examples/process.html>`_
* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for
  both C and Python sections.
  See some `cPyMemTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html>`_
  and a `technical note on cPyMemTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html>`_.
* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and
  report how much memory was allocated and by whom.
  See some `DTrace examples <https://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html>`_
  and a `technical note on DTrace <https://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html>`_.
* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.
  This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.
  This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
  See some `trace_malloc examples <https://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html>`_
* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of
  memory before and after code execution and report the significant differences of the Python small object allocator.
  See some `debug_malloc_stats examples <https://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html>`_


Tool Characteristics
======================

Each tool can be characterised by:

- *Memory Granularity*: In how much detail is a memory change is observed.
  An example of *coarse* memory granularity is measuring the
  `Resident Set Size <https://en.wikipedia.org/wiki/Resident_set_size>`_ which is normally in chunks of 4096 bytes.
  An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.
- *Execution Granularity*: In how much code detail is the memory change observed.
  An example of *coarse* execution granularity is measuring the memory usage every second.
  An example of *fine* execution granularity is recording the memory usage every Python line.
- *Memory Cost*: How much extra memory the tool needs.
- *Execution Cost*: How much the execution time is increased.

Clearly there are trade-offs between these depending on the problem you are trying to solve.

.. list-table:: **Tool Characteristics**
   :widths: 15 30 30 30 30
   :header-rows: 1

   * - Tool
     - Memory Granularity
     - Execution Granularity
     - Memory Cost
     - Execution Cost
   * - ``process``
     - RSS (total Python and C memory).
     - Regular time intervals.
     - Near zero.
     - Near zero.
   * - ``cPyMemTrace``
     - RSS (total Python and C memory).
     - Per Python line, Python function and C function call.
     - Near zero.
     - x10 to x20.
   * - DTrace
     - Every ``malloc()`` and ``free()``.
     - Per function call and return.
     - Minimal.
     - x90 to x100.
   * - ``trace_malloc``
     - Every Python object.
     - Per Python line, per function call.
     - Significant but compensated.
     - x900 for small objects, x6 for large objects.
   * - ``debug_malloc_stats``
     - Python memory pool.
     - Snapshots the CPython memory pool either side of a block of code.
     - Minimal.
     - x2000+ for small objects, x12 for large objects.

Package Metadata
=========================

.. image:: https://img.shields.io/pypi/v/pymemtrace.svg
        :target: https://pypi.python.org/pypi/pymemtrace

.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg
        :target: https://travis-ci.org/paulross/pymemtrace

.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest
        :target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg
     :target: https://pyup.io/repos/github/paulross/pymemtrace/
     :alt: Updates
    

Licence
-----------------------

Python memory tracing.

* Free software: MIT license
* Documentation: https://pymemtrace.readthedocs.io.
* Project: https://github.com/paulross/pymemtrace.

Credits
-----------------

Phil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, of ``cPyMemTrace`` in pure Python.

This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.

.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage



=======
History
=======

0.2.0 (2024-11-17)
---------------------

* cPyMemTrace:
    * Add P/T, stack depth and python version to log file name, example:
      "20241107_195847_62264_P_0_PY3.13.0b3.log"
    * Add stacking of trace/profile functions with linked list of tTraceFileWrapperLinkedList.
    * Add an option to log to a specific file.
    * Add an API write_to_log() to inject text into the log file.
    * Add an optional message to the log file in cPyMemTrace.
    * Add Python API to get log file being written to by cPyMemTrace.
    * Bug fixes in cPyMemTrace.c
    * Safety fix for file path name lengths.
    * Fix for log files where '#' was being concatenated.

0.1.7 (2024-09-12)
------------------

* Minor fix for a single test.

0.1.6 (2024-09-11)
------------------

* Add support for Python versions 3.12, 3.13. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.

0.1.5 (2023-06-21)
------------------

* Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.

0.1.4 (2022-03-19)
------------------

* Fix Linux build.

0.1.3 (2022-03-17)
------------------

* Fix some tests.

0.1.2 (2022-03-17)
------------------

* Fix source distribution that had missing headers.

0.1.1 (2020-11-17)
------------------

* Add cPyMemTrace the C level profiler.
* Add DTrace scripts for low level tracing.
* Add debug_malloc_stats the wrapper around sys._debugmallocstats.
* Add process from the TotalDepth project.
* Add redirect_stdout for debug_malloc_stats.
* Add trace_malloc, a wrapper around the tracemalloc module.
* Includes extensive documentation and performance measurement.
* First release on PyPI.

0.1.0 (2017-12-04)
------------------

* Initial idea and implementation, never released.