django-apm 0.3.0
A fully featured APM for django using django
License: MIT
Requires Python: >=3.9

# django-apm

A fully featured APM for django using django.

## Features:

- Know each view is getting more requests/errors;
- Keep track of response times;
- Registers errors and notify you/your team on slack/discord with the exception, traceback and logs;
- Keep track of requests headers/query and payloads (on errors), making it easy to reproduce them in a separate environment.
- Notify your team using celery if desired;

Notifications example:
On slack:
<img src="docs/examples/slack-notification.png">
On discord:
<img src="docs/examples/discord-notification.png">

## Installation

Install the package using your favorite packaging tool: pip / poetry / pdm, etc.

1.  **Add `djapm.apm.apps.ApmConfig` to `INSTALLED_APPS` in your django settings. Make sure to add it before `django.contrib.admin`.** Otherwise, the dashboard link will not show up in Django admin.
    If you're going to use the `apm_api_view`: Don't forget to add `rest_framework` to your `INSTALLED_APPS`.
    It also requires the `django.contrib.sites` app installed. It must come before the `django.contrib.admin` for the dashboard link show up.

2.  **Adding the middlewares**

    - Adding the ApmMetricsMiddleware middleware (optional).
      If you want to track the requests timing, add the `djapm.apm.middlewares.ApmMetricsMiddleware` to `MIDDLEWARE` in your django settings. Note that this will store into your database **all** requests that use the decorator `apm_api_view` (This decorator is explained later), no cleanup is done. Is up to you to clean the registers periodically if desired. It's better to keep this middleware the **closest to the top** of your `MIDDLEWARE` as possible , this way the `ellapsed` time is closer to the reality.

    - Adding the ErrorTraceMiddleware (optional).
      If you want to track errors and receive notifications, add the `djapm.apm.middlewares.ErrorTraceMiddleware` to `MIDDLEWARE` in your django settings. We recommend that you keep this middleware **closest to the bottom** of your `MIDDLEWARE` as possible. This middleware will notify any exception raised on your view to all integrations added (the setup is explained later).

3.  **Including the URLs**

    `djapm` comes with a dashboard that's accessible through the django-admin. It only allows superusers to view it. To enable the dashboard include the following in your root `urls.py` file:

    ```python
    from django.urls import include

    urlpatterns = [
      # Your other patterns...
      path("apm/", include("djapm.apm.urls")),
    ]
    ```

    Now go to the admin, you should be able to see the dashboard link.
    <img src="./docs/examples/admin-apm-app-list.png">

    The dashboard shows the data collected from the middlewares. It uses `chart.js` to render the graphics. The data is get from the database via API. Example:
    <img src="./docs/examples/dashboard.png">

4.  **Upgrading your views**
    django-apm comes with 3 decorators: `apm_api_view`, `apm_view` and `apm_admin_view`. Also, it comes with two ClassBasedViews: `ApmView` and `ApmAPIView`. Also there's a ModelAdmin for tracking POST requests: `ApmModelAdmin`. Each of these adds some attributes to the `request`, they are:

    - `id` (`str`): A string UUID4;
    - `logger` (`logging.Logger`): A logger that you can use to log to the sdout and keep track of all logs emitted.
    - `_log_handler` (`djapm.apm.log.ApmStreamHandler`): We use this handler so we can capture all logs emitted by your view; **Do not use this directly**.
    - `_json` (`Any`): An alias to `rest_framework.Request.data` we store this because it's only available at the view, and if not stored, we wouldn't be able to track the payload in the middleware.

    If you enabled the `ApmMetricsMiddleware`, this request and response will be saved to the database. If you enabled the `ErrorTraceMiddleware` any errors uncaught raised by your view will be notified to you. This does not includes the `rest_framework.serializers.ValidationError`, since `rest_framework` itself already handles this exception.

    So, let's get started with each decorator:

    - `apm_api_view`:
      This decorator is an extension of the`rest*framework.decorators.api_view` decorator, so you don't need to wrap your view with the`rest_framework` decorator.

      _Example api view_:

      ```python
      from djapm.apm.decorators import apm_api_view
      from djapm.apm.types import ApmRequest

      @apm_api_view(["GET"])
      def your_view(request: ApmRequest, **kwargs):
           logger = request.logger
           req_id = request.id
           ...
      ```

      **What's changed from the `rest_framework` function-based view:**
      The `request` parameter is now of the type `ApmRequest`, it inherits from `rest_framework.request.Request` so you still have access to all it's attributes.
      You receive a `logger` and a `id` that you can use. The `logger` was created just before your view using a default, if you need to use a different logger you can pass the logger name in the decorator.

    - `apm_view`:
      Add this decorator to all your django regular function-based view's that you want to keep track of.

      _Example django view_:

      ```python
      from djapm.apm.decorators import apm_view
      from djapm.apm.types import PatchedHttpRequest

      @apm_view()
      def your_django_regular_view(request: PatchedHttpRequest, **kwargs):
          ...
      ```

    - `apm_admin_view`:
      Add this decorator to all your django regular admin view's that you want to keep track of.

      _Example django admin view_:

      ```python
      from django.contrib import admin
      from djapm.apm.decorators import apm_admin_view
      from djapm.apm.types import PatchedHttpRequest

      class YourAdmin(admin.ModelAdmin):
          @apm_admin_view()
          def your_custom_admin_view(self, request: PatchedHttpRequest, **kwargs):
              ...

          # don't forget to override the `get_urls` the same way as when you did with your custom view.
      ```

    - The class-based-view (CBV) version:

      If you want to keep track of your CBVs. Inherit the `ApmView` for your regular django views, or the `ApmAPIView` for your DRF's API views. This way we can add the required attributes before your view gets called.

      _Example django CBV view_:

      ```python
      from django.views.generic import CreateView # or any other view
      from djapm.apm.views import ApmView
      from djapm.apm.types import PatchedHttpRequest

      class YourCreateView(CreateView, ApmView):
          ...
          def post(self, request: PatchedHttpRequest, **kwargs):
              ...
      ```

      _Example DRF CBV view:_
      ```python
      from djapm.apm.generics import ApmCreateAPIView  # or any other view
      from djapm.apm.types import ApmRequest


      class YourAPICreateView(ApmCreateAPIView):
          ...
          def create(self, request: ApmRequest, *args, **kwargs):
              ...
      ```

    - The Django Admin POST requests:
      If you want to keep track of your admin model Creations, inherit from `ApmModelAdmin` instead of the regular `admin.ModelAdmin`. Example:

      ```python
      from django.contrib import admin
      from djapm.apm.admin import ApmModelAdmin
      from yourapp.models import AModel

      @admin.register(AModel)
      class YourModelAdmin(ApmModelAdmin):
        ...
      ```

5.  **Running the migrations:**

    Since we rely on the database, you must migrate your database:
    `python manage.py migrate`. This will create these models:

    - `ApiRequest` - Keep tracks of requests, including headers, query parameters;
    - `ApiResponse` - Keep tracks of responses, including status_code, ellapsed time.
    - `ErrorTrace` - Keep tracks of errors on your views.
    - `RequestLog` - Keep track of all logs that were logged from the `request.logger`;
    - `Integration` - Keep track of all your integrations;
    - `NotificationReceiver` - Keep track of all notifications receivers of an integration.

6.  **Create an Integration**
    Now go to the admin, you should now see the models registered there.
    Go to integrations and create at least one integration. Current supported integrations:

    - Slack;
      Uses the Slack App key to send messages through to it's API. Here's a [guide](https://api.slack.com/authentication/basics) on how to create your app and get the key. Scopes required: `channels:read`, `chat:write`. After you get this configured. Install the slack app into the workspace that should send messages. Later, allow/invite the bot to a conversation/channel. When creating the `Integration` on the admin, the slack integration supports both the `name` and `id` of a channel conversation.
      The `name` of the channel is the visible name that appear on slack, note that using the name will require that we fetch the `id` on the slack api.
      The `id` of the channel can be retrieved on Slack. Click with the mouse's right button and click on `Copy` -> `Copy Link`, the `id` is the last part of the URL.
      The same applies for a user.
      <img src="./docs/examples/slack-integration.png">
    - Discord;
      Uses a Discord Bot account to send messages through it's API. Here's a [guide](https://discord.com/developers/docs/intro) on how to create your Bot account. Required scopes: `guilds`, `SEND_MESSAGES`.
      When creating the `Integration` on the admin, the discord integration supports both the `name` and `id` of a channel conversation.
      The `name` of the channel is the visible name that appear on discord, note that using the name will require that we fetch the `id` on the discord api. So we need to: Get your Bot `guilds`, for each `guild` get it's `channels` and lookup for a channel with the exact name that was provided. The message is sent to the exact first channel found on any guild, so use a specific name.
      The `id` of the channel can be retrieved on Discord. Click with the mouse's right button and click on `Copy Link`, the `id` is the last part of the URL.
      The same applies for a user.
      <img src="./docs/examples/discord-integration.png">

      That's it! You're setup to keep track of your Requests. Also, take a look into the source code `example` folder, that contains a django-project that uses the `djapm` package.

7.  **Optional settings**

    `django-apm` has a set of optional configurations, that can be set on your django settings module. Checkout them below:

    - `APM_DEFAULT_LOGGER_NAME`: The default logger name that will be used if None is supplied to the `apm_view_decorator`. Defaults to: `apm_api_view`;
    - `APM_REQUEST_SAVE_HEADERS`: Boolean that when set to `True` will save the request headers. Defaults to `True`;
    - `APM_REQUEST_SAVE_QUERY_PARAMETERS`: Boolean that when set to `True` will save the request query parameters as json. Defaults to `True`;
    - `APM_REQUEST_SAVE_QUERY_STRING`: Boolean that when set to `True` will save the request raw query string. Defaults to `True`;
    - `APM_NOTIFY_USING_CELERY`: Boolean that when set to `True` will dispatch a celery task when notificating. Since the process of notificating Integration can take a long time, we suggest you to set this to `True`. Defaults to `False`.
    - `APM_NOTIFY_ON_DEBUG_TRUE`: Boolean that when set to `True` will notify errors even when `DEBUG=True`. Defaults to `False`.
    - `APM_USE_DATABASE`: String representing a key to the `DATABASES` django setting. Defaults to the django-default "default". Useful for changing which database apm will use to store it's data.

## Storage considerations

Since `djapm` uses the database to register, get metrics, this may lead to a lot of storage being used. `djapm` doesn't do cleanups, if you use celery-beat it may be useful to have a scheduled task to do the cleanup. Most of the data displayed on the dashboard is from the last 7 days.

You also may find useful to use a separate database for this metrics, errors. For that use the `APM_USE_DATABASE` setting.

## Ellapsed time considerations

The ellapsed time displayed/registered is not precise from what your clients may be having. Since we only start keeping track of the time when the request first enters the `ApmMetricsMiddleware`.

## Project Future

- Include support for ViewSets.