edx-django-utils 8.0.1
EdX utilities for Django Application development.
Keywords: Django edx

edx-django-utils
================

|pypi-badge| |ci-badge| |codecov-badge| |doc-badge| |pyversions-badge|
|license-badge| |status-badge|

EdX utilities for Django Application development.

Note that some utilities may warrant their own repository. A judgement call
needs to be made as to whether code properly belongs here or not. Please
review with the Architecture Team if you have any questions.

Purpose
-------

This repository includes shared utilities for:

* `Cache Utilities`_: Includes a RequestCache and a TieredCache.

* `Django User and Group Utilities`_: Includes user and group utilities.

* `IP Address Utilities`_: Utilities for handling request IP addresses.

* `Logging Utilities`_: Includes log filters and an encrypted logging helper.

* `Monitoring Utilities`_: Includes Middleware and utilities for enhanced monitoring.
  Supports several monitoring backends.

* `Plugin Infrastructure`_: Enables enhanced Django Plugin capabilities.

* `Security Utilities`_: Includes a middleware to add CSP response headers.

* `Data Generation`_: Management command for generating Django data based on model factories.

.. _Cache Utilities: edx_django_utils/cache/README.rst

.. _Django User and Group Utilities: edx_django_utils/user/README.rst

.. _IP Address Utilities: edx_django_utils/ip/README.rst

.. _Logging Utilities: edx_django_utils/logging/README.rst

.. _Monitoring Utilities: edx_django_utils/monitoring/README.rst

.. _Plugin Infrastructure: edx_django_utils/plugins/README.rst

.. _Security Utilities: edx_django_utils/security/README.rst

.. _Data Generation: edx_django_utils/data_generation/README.rst

Documentation
-------------

The full documentation is in the docs directory, and is published to https://edx-django-utils.readthedocs.org.

Getting Started with Development
--------------------------------

Please see the Open edX documentation for `guidance on Python development <https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html>`_ in this repo.

Design Pattern followed by packages
-----------------------------------

All tools in edx_django_utils should expose their public api in their __init__.py files. This entails adding to __init__.py all functions/classes/constants/objects that are intended to be used by users of library.

Getting Help
------------

If you're having trouble, we have discussion forums at
`discuss.openedx.org <https://discuss.openedx.org>`_ where you can connect with others in the
community.

Our real-time conversations are on Slack. You can request a `Slack
invitation`_, then join our `community Slack workspace`_.

For anything non-trivial, the best path is to `open an issue`__ in this
repository with as many details about the issue you are facing as you
can provide.

__ https://github.com/openedx/django-config-models/issues

For more information about these options, see the `Getting Help`_ page.

.. _Slack invitation: https://openedx.org/slack
.. _community Slack workspace: https://openedx.slack.com/
.. _Getting Help: https://openedx.org/getting-help

How To Contribute
-----------------

Contributions are very welcome.

Please read `How To Contribute <https://github.com/openedx/.github/blob/master/CONTRIBUTING.md>`_ for details.

PR description template should be automatically applied if you are sending PR from github interface; otherwise you
can find it it at `PULL_REQUEST_TEMPLATE.md <https://github.com/openedx/edx-django-utils/blob/master/.github/PULL_REQUEST_TEMPLATE.md>`_

Issue report template should be automatically applied if you are sending it from github UI as well; otherwise you
can find it at `ISSUE_TEMPLATE.md <https://github.com/openedx/edx-django-utils/blob/master/.github/ISSUE_TEMPLATE.md>`_

This project is currently accepting all types of contributions, bug fixes, security fixes, maintenance work, or new features. However, please make sure to have a discussion about your new feature idea with the maintainers prior to beginning development to maximize the chances of your change being accepted. You can start a conversation by creating a new issue on this repo summarizing your idea.

Open edX Code of Conduct
------------------------
All community members are expected to follow the `Open edX Code of Conduct`_.

.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/

People
------
The assigned maintainers for this component and other project details may be
found in `Backstage`_. Backstage pulls this data from the ``catalog-info.yaml``
file in this repo.


.. _Backstage: https://backstage.openedx.org/catalog/default/component/edx-django-utils

Reporting Security Issues
-------------------------

Please do not report security issues in public. Please email security@openedx.org.

License
-------

The code in this repository is licensed under the Apache License, Version 2.0, unless
otherwise noted.

Please see ``LICENSE.txt`` for details.


.. |pypi-badge| image:: https://img.shields.io/pypi/v/edx-django-utils.svg
    :target: https://pypi.python.org/pypi/edx-django-utils/
    :alt: PyPI

.. |ci-badge| image:: https://github.com/openedx/edx-django-utils/workflows/Python%20CI/badge.svg?branch=master
    :target: https://github.com/openedx/edx-django-utils/actions?query=workflow%3A%22Python+CI%22
    :alt: CI

.. |codecov-badge| image:: http://codecov.io/github/edx/edx-django-utils/coverage.svg?branch=master
    :target: http://codecov.io/github/edx/edx-django-utils?branch=master
    :alt: Codecov

.. |doc-badge| image:: https://readthedocs.org/projects/edx-django-utils/badge/?version=latest
    :target: http://edx-django-utils.readthedocs.io/en/latest/
    :alt: Documentation

.. |pyversions-badge| image:: https://img.shields.io/pypi/pyversions/edx-django-utils.svg
    :target: https://pypi.python.org/pypi/edx-django-utils/
    :alt: Supported Python versions

.. |license-badge| image:: https://img.shields.io/github/license/edx/edx-django-utils.svg
    :target: https://github.com/openedx/edx-django-utils/blob/master/LICENSE.txt
    :alt: License

.. |status-badge| image:: https://img.shields.io/badge/Status-Maintained-brightgreen
    :alt: Maintenance status


Change Log
==========

..
   All enhancements and patches to edx_django_utils will be documented
   in this file.  It adheres to the structure of https://keepachangelog.com/ ,
   but in reStructuredText instead of Markdown (for ease of incorporation into
   Sphinx documentation and the PyPI description).

   This project adheres to Semantic Versioning (https://semver.org/).

.. There should always be an "Unreleased" section for changes pending release.

Unreleased
----------

8.0.1 - 2025-09-29
------------------
* Stop using deprecated newrelic function calls that were removed in newrelic 11.0.0 (use newer names instead)

8.0.0 - 2025-05-22
------------------
* **BREAKING CHANGE**: ``newrelic`` removed from dependencies. Users of New Relic should install the package separately in their services. `See relevant DEPR <https://github.com/openedx/public-engineering/issues/360>`_.

7.4.0 - 2025-04-10
------------------
* Same as 7.3.0
* 7.3.0 tag was created but PyPI release job had failed

7.3.0 - 2025-04-09
------------------
Added
~~~~~
* Added support for Django 5.2

7.2.0 - 2025-02-26
------------------
Removed
~~~~~~~
* Removed deprecated monitoring middleware Classes

7.1.0 - 2024-12-05
------------------
Added
~~~~~
* Added signals monitoring_support_process_request, monitoring_support_process_response, and monitoring_support_process_exception to the MonitoringSupportMiddleware to enable plugging in monitoring capabilities.

7.0.1 - 2024-11-21
------------------
Fixed
~~~~~
* Fix bug where code owner custom attributes were being defined, even when the CODE_OWNER_MAPPINGS setting was not defined.

7.0.0 - 2024-10-16
------------------
Removed
~~~~~~~
* Remove unused ``background_task`` monitoring function.
* Remove ``get_current_transaction`` (used internally only) from the public API.

[6.1.0] - 2024-10-15
---------------------
Changed
~~~~~~~
* Added Datadog implementation of  ``set_monitoring_transaction_name`` and refactored the functionality.

[6.0.0] - 2024-10-09
---------------------
Added
~~~~~
* Added support for python3.12
* Dropped support for python<3.12 versions

[5.16.0] - 2024-09-27
---------------------
Added
~~~~~
* Added a new method to backends for ``tag_root_span_with_error`` and added Datadog implementation of the functionality.
* Uses the new method to tag the root span when processing exceptions.

Changed
~~~~~~~
* Renamed ``CachedCustomMonitoringMiddleware`` to ``MonitoringSupportMiddleware`` and deprecated the old name. It will be removed in a future release.

[5.15.0] - 2024-07-29
---------------------
Added
~~~~~
* Added Datadog implementation of ``function_trace`` and allowed implementation to be configurable.

[5.14.2] - 2024-05-31
---------------------
Fixed
~~~~~
* FrontendMonitoringMiddleware now updates the Content-Length header if its already set.
* FrontendMonitoringMiddleware will now be enabled once waffle switch named ``edx_django_utils.monitoring.enable_frontend_monitoring_middleware`` is enabled.

[5.14.1] - 2024-05-22
---------------------
Fixed
~~~~~
* Added default value while getting content-type header to avoid KeyError.

[5.14.0] - 2024-05-22
---------------------
Added
~~~~~
* Added middleware named ``FrontendMonitoringMiddleware`` for inserting frontend monitoring HTML script tags to response, configured by new Django setting ``OPENEDX_TELEMETRY_FRONTEND_SCRIPTS``.

[5.13.0] - 2024-04-30
---------------------
Added
~~~~~
* Initial support for sending monitoring data to OpenTelemetry collector or Datadog agent, configured by new Django setting ``OPENEDX_TELEMETRY``. See monitoring README for details.

[5.12.0] - 2024-03-29
---------------------
Added
~~~~~
* Added support for ``Python 3.11``

[5.11.0] - 2024-03-06
---------------------
Added
~~~~~
* Added support for ``Python 3.12``

Removed
~~~~~~~
* Dropped support for ``Django 3.2``

[5.10.1] - 2024-01-17
---------------------

Added
~~~~~
* Added manufacture_data management command

[5.9.0] - 2023-11-27
--------------------

Removed
~~~~~~~
* Removed ``edx_django_utils.cache.disable_forced_cache_miss_for_none`` which was added in ``5.7.0``.

[5.8.0] - 2023-11-03
--------------------

Changed
~~~~~~~
* Adjusted ``get_plugin_apps`` to log at info level rather than debug and with more detail, though with a comment that this may not actually end up logging.

[5.7.0] - 2023-08-04
--------------------

Added
~~~~~
* Support added for Django 4.2

Fixed
~~~~~
* Fixed bug where None was not properly being stored by TieredCache.
  For backward compatibility, ``edx_django_utils.cache.disable_forced_cache_miss_for_none`` waffle switch has
  been added, which defaults to the old broken behavior of treating None as a cache miss.

[5.6.0] - 2023-07-24
--------------------

Changed
~~~~~~~
* Updated and renamed new_relic_nrql_search to search in text widgets as well as NRQL queries

[5.5.0] - 2023-06-01
--------------------

Changed
~~~~~~~
* Switched to ``sphinx-book-theme`` as the new standard theme across all Open
  edX repos.  See https://github.com/openedx/edx-sphinx-theme/issues/184 for
  more details.
* CookieMonitoringMiddleware will now remove cookies based on a ``COOKIE_PREFIXES_TO_REMOVE`` setting


[5.4.0] - 2023-04-12
--------------------

Added
~~~~~

* Added Content-Security-Policy response header middleware under ``security/csp``

[5.3.0] - 2023-03-17
--------------------

Fixed
~~~~~

* Report both specified and existing email when refusing to create a user

[5.2.0] - 2022-10-06
--------------------

Added
~~~~~

* Added a wrapper for background_task in monitoring.

[5.1.0] - 2022-09-19
--------------------

Added
~~~~~

* Utilities for safely determining the IP address of a request: ``edx_django_utils.ip`` (moved from edx-platform)

[5.0.1] - 2022-09-09
--------------------

Changed
~~~~~~~

* License has been changed from AGPL v3 to Apache v2 to reflect existing policies

[5.0.0] - 2022-05-19
--------------------

Changed
~~~~~~~

* Corrupt cookie logging:

  * Make independent of other cookie logging; no longer needs to meet cookie size threshold or sampling rate.
  * **Breaking change**, although low impact: Setting name changed from ``UNUSUAL_COOKIE_SAMPLING_PUBLIC_KEY`` to ``UNUSUAL_COOKIE_HEADER_PUBLIC_KEY``.
  * New setting ``UNUSUAL_COOKIE_HEADER_LOG_CHUNK`` helps avoid truncated (non-decryptable) messages by splitting the output across multiple log messages.

[4.8.1] - 2022-05-06
--------------------

Added
~~~~~

* Added ability to log headers when a corrupted cookie is detected in a large (or sampled) cookie header

[4.8.0] - 2022-05-06
--------------------

Bad version -- tag does not match package version, not released to PyPI. Released as 4.8.1 instead.

[4.7.0] - 2022-05-05
--------------------

Added
~~~~~

* Added ``encrypt_for_log`` logging helper and ``log-sensitive`` CLI command for encrypted logging of sensitive information

[4.6.0] - 2022-03-16
--------------------

Added
~~~~~

* Added ``CookieMonitoringMiddleware`` for monitoring cookie header sizes and cookie sizes.

[4.5.0] - 2022-01-31
--------------------

Removed
~~~~~~~

* Removed Django22, 30 and 31 from CI

[4.4.2] - 2022-01-24
--------------------

Fixed
~~~~~

* No longer clear the ``RequestCache`` during the exception-handling phase (wait until response phase)

  * It turns out all the ``process_exception`` methods get called until one returns a response, and only *then* do the ``process_response`` methods start getting called. The result was that on exception, some middlewares were unable to use RequestCache'd values in their response phase.

Updated
~~~~~~~

* Replaced usage of 'django.conf.urls' with 'django.urls'

[4.4.1] - 2021-12-17
--------------------

Updated
~~~~~~~

* Replaced usage of 'django.conf.urls.url()' with 'django.urls.re_path()'

[4.4.0] - 2021-09-02
--------------------

Added
~~~~~

* Added ``DeploymentMonitoringMiddleware`` to record ``Python`` and ``Django`` versions in NewRelic with each transaction.

[4.3.0] - 2021-07-20
--------------------

Added
~~~~~

* Added user and group management utilities.

[4.2.0] - 2021-07-14
--------------------

Added
~~~~~

* Added support for Django 3.1 and 3.2

[4.1.0] - 2021-06-01
--------------------

Added
~~~~~

* Added mixin for a custom Django admin class which disables CRUD operation on the admin's model.

Added
~~~~~

* Script new_relic_nrql_search.py to search the NRQL in New Relic alert policies and dashboards using a supplied regex.

[4.0.0] - 2021-05-03
--------------------

Removed
~~~~~~~

* Removed the old location of ``CodeOwnerMonitoringMiddleware``. It had moved in a past commit. Although technically a breaking change, all references in the Open edX platform have already been updated to point to the new location.

Added
~~~~~

* Added new ``code_owner_theme`` and ``code_owner_squad`` custom attributes. This is useful in cases where the ``code_owner`` combines a theme and squad name, because monitoring can instead reference ``code_owner_squad`` to be resilient to theme name updates. For the decision doc, see edx_django_utils/monitoring/docs/decisions/0004-code-owner-theme-and-squad.rst.

Updated
~~~~~~~

* Misconfigurations of CODE_OWNER_MAPPINGS will now fail fast, rather than just logging. Although technically a breaking change, if CODE_OWNER_MAPPINGS is in use, it is probably correctly configured and this change should be a no-op.

[3.16.0] - 2021-03-24
---------------------

Added
~~~~~

* Added ``pluggable_override`` decorator.


[3.15.0] - 2021-03-02
---------------------

* Added chunked_queryset utility.

[3.14.0] - 2020-12-15
---------------------

Removed
~~~~~~~

* Dropped support for Python 3.5.


[3.13.0] - 2020-11-18
---------------------

Added
~~~~~

* Added record_exception to monitor caught exceptions.

Updated
~~~~~~~

* Added additional details to the `deprecated_monitoring_utils` custom attribute values to make it simpler to track down usage.

[3.12.0] - 2020-11-17
---------------------

Added
~~~~~

* Added set_code_owner_attribute decorator for use with celery tasks.
* Added set_code_owner_attribute_from_module as an alternative to the decorator.

Updated
~~~~~~~

* Cleaned up some of the code owner middleware code. In doing so, renamed custom attribute code_owner_path_module to code_owner_module. This may affect monitoring dashboards. Also slightly changed when error custom attributes are set.

[3.11.0] - 2020-10-31
---------------------

Added
~~~~~

* Added ADR 0004-public-api-and-app-organization.rst to explain a new app organization, which makes use of the public API more consistent.

Updated
~~~~~~~

* Applied the new app organization described in th ADR to the monitoring Django app.
* Moved CachedCustomMonitoringMiddleware, CodeOwnerMonitoringMiddleware, and MonitoringMemoryMiddleware to the public API.

Deprecated
~~~~~~~~~~

* Deprecated the old locations of CachedCustomMonitoringMiddleware, CodeOwnerMonitoringMiddleware, and MonitoringMemoryMiddleware.
* Deprecated various methods from modules that were always meant to be used from the public API.

  * accumulate
  * increment
  * set_custom_attribute
  * set_custom_attributes_for_course_key

* Added additional custom attributes for deprecated classes and methods to make them safer to retire.

.. note::

  Some method implementations that were available in the public API were moved without adding a deprecated equivalent. These were not found when searching, so hopefully they are only used via the public API, which did not change. This includes functions in ``transactions.py`` and ``code_owner/utils.py``.

Removed
~~~~~~~

* Removed the middleware ordering checks. This is not a typical Django feature and it is painful when refactoring.

[3.10.0] - 2020-10-28
---------------------

Added
~~~~~

* Added logging filter classes for users and remote IP addresses to be used by all IDAs. These were moved here from edx-platform.

[3.9.0] - 2020-10-21
--------------------

Updated
~~~~~~~

* Exposed existing get_code_owner_from_module via the public api.
* Fixed get_code_owner_from_module to not require a call to is_code_owner_mappings_configured beforehand.
* Set the existing code_owner_path_module custom attribute, even for cases where the transaction name was used, rather than the view module.
* Refactor code owner setting processing.

[3.8.0] - 2020-08-31
--------------------

Updated
~~~~~~~

* Renamed "custom metric" to "custom attribute" throughout the monitoring library. This decision can be read about in the ADR 0002-custom-monitoring-language.rst.  The following have been deprecated:

  * set_custom_metric (use set_custom_attribute)
  * set_custom_metrics_for_course_key (use set_custom_attributes_for_course_key)
  * MonitoringCustomMetricsMiddleware (use CachedCustomMonitoringMiddleware)
  * CachedCustomMonitoringMiddleware.accumulate_metric (use CachedCustomMonitoringMiddleware.accumulate_attribute)

    * This wasn't meant to be used publicly, but was deprecated just in case.

  * CodeOwnerMetricMiddleware (use CodeOwnerMonitoringMiddleware)

[3.7.4] - 2020-08-29
--------------------

* Fix to custom monitoring accumulate to actually accumulate rather than overwrite.

[3.7.3] - 2020-08-12
--------------------

Updated
~~~~~~~

* Upgrade psutil to latest version

[3.7.2] - 2020-08-10
--------------------

Updated
~~~~~~~

* Added missing classes to plugins public api. See ``plugins.__init__.py`` for latest api.
* Updated plugin method names to be more descriptive. See ``plugins.__init__.py`` for latest.

.. note:: Although these changes are backwards incompatible, they are being added as a bug fix because plugins code release (3.7.0) is not yet in use.

[3.7.1] - 2020-08-10
--------------------

Updated
~~~~~~~

* Exposing all public functions in edx_django_utils/plugins directory in its __init__.py file.
    * this was done to keep inline with standard/pattern used in other packages in edx_django_utils

[3.7.0] - 2020-08-10
--------------------

Added
~~~~~

* Adding Plugin infrastructure
    * Allows IDAs to use plugins

[3.6.0] - 2020-08-04
--------------------

Added
~~~~~

* Improved documentation for CodeOwnerMetricMiddleware, including a how_tos/add_code_owner_custom_metric_to_an_ida.rst for adding it to a new IDA.
* Added ignore_transaction monitoring utility to ignore transactions we don't want tracked.

Updated
~~~~~~~

* Moved transaction-related monitoring code into it's own file. Still exposed through `__init__.py` so it's a non-breaking change.

[3.5.0] - 2020-07-22
--------------------

Updated
~~~~~~~

* Added a catch-all capability to CodeOwnerMetricMiddleware when CODE_OWNER_MAPPINGS includes a '*' as a team's module. The catch-all is used only if there is no other match.

[3.4.0] - 2020-07-20
--------------------

Added
~~~~~

* Added get_current_transaction for monitoring that returns a transaction object with a name property.

Updated
~~~~~~~

* Updated CodeOwnerMetricMiddleware to use NewRelic's current transaction for cases where resolve() doesn't work to determine the code_owner, like for Middleware.

[3.3.0] - 2020-07-16
--------------------

Added
~~~~~

* CodeOwnerMetricMiddleware was moved here (from edx-platform) in order to be able to take advantage of the ``code_owner`` metric in other IDAs. For details on this decision, see the `ADR for monitoring code owner`_. See the docstring for more details on usage.

.. _ADR for monitoring code owner: https://github.com/openedx/edx-django-utils/blob/master/edx_django_utils/monitoring/docs/decisions/0001-monitoring-by-code-owner.rst

[3.2.3] - 2020-05-30
--------------------
* Removed ceninusepy3 usage.

[3.2.2] - 2020-05-04
--------------------
* Added support for python 3.8 and dropped support for Django versions older than 2.2

[3.2.1] - 2020-04-17
--------------------

Changed
~~~~~~~

* imported get_cache_key in cache/__init__.py.

[3.2.0] - 2020-04-09
--------------------

Added
~~~~~

* Added get_cache_key utility.

[2.0.1] - 2019-10-09
--------------------

Changed
~~~~~~~

* Fixed: Updated function tracing to accomodate changes in New Relic's 5.x Agent.

[2.0.0] - 2019-07-07
--------------------

Changed
~~~~~~~

* Converted Middleware (from old style MIDDLEWARE_CLASSES to MIDDLEWARE).
* Removed support for Django versions < 1.11

[1.0.1] - 2018-09-07
--------------------

Changed
~~~~~~~

* Fixed: RequestCache now properly uses thread.local.
* Fixed: CachedResponse.__repr__ now handles unicode.

[1.0.0] - 2018-08-28
--------------------

Added
~~~~~~~

* Add ``data`` dict property to better match legacy RequestCache interface.

Changed
~~~~~~~

* Change is_hit/is_miss to is_found.

[0.5.1] - 2018-08-17
--------------------

Changed
~~~~~~~

* Fixed bug in TieredCacheMiddleware dependency declaration.

[0.5.0] - 2018-08-16
--------------------

Changed
~~~~~~~

* Restored Python 3 support.
* Refactor/clean-up, including Middleware dependency checking.
* Docs updates and other cookiecutter updates.

[0.4.1] - 2018-08-10
--------------------

Changed
~~~~~~~

* Split out TieredCacheMiddleware from RequestCacheMiddleware.

[0.4.0] - 2018-08-10
--------------------

Changed
~~~~~~~

* Rename CacheUtilsMiddleware to RequestCacheMiddleware.

[0.3.0] - 2018-08-02
--------------------

Removed
~~~~~~~

* Temporarily dropped Python 3 support to land this.

[0.2.0] - 2018-08-01
--------------------

Added
~~~~~

* Added cache and monitoring utilities.


[0.1.0] - 2018-07-23
--------------------

Added
~~~~~

* First release on PyPI.