Logging in Python
Tags: python, logging
Published: 2022-10-14 18:38:11

*This article was originally written by [Ravgeet Dhillon](https://www.honeybadger.io/blog/python-logging/#authorDetails) on the [Honeybadger Developer Blog](https://www.honeybadger.io/blog/python-logging/).*

When an application runs, it performs a tremendous number of tasks. A simple to-do app can have tons of tasks like - user logins, creating to-dos, updating to-dos, deleting to-dos, and duplicating to-dos. These tasks can result in success or may end up with some errors. Hence, there is a need to monitor events happening and analyze them to identify bottlenecks in the performance of the application. This is where **logging** is useful.

In this article, you'll learn how to create logs in a Python application using the [Python logging module](https://docs.python.org/3/library/logging.html). Logging can help Python developers of all experience levels develop and analyze an application's performance more quickly.

## What is Logging?

Logging is the process of storing the details about events that happen in an application. Logs provide an additional way to check out the flow that an application is growing through. It is the historical record of the state of the application.

For instance, if an application crashes, it becomes difficult to track the issue without any details of what happened before the crash. In such a case, logging provides a log trace, which can help developers detect the cause of the issue by going through the logs and recreating the actual scenario on their machine. With a properly set up logging system, you can detect the cause of errors to the accuracy level of the line number.

## When and Why Should Developers Use Logging?

If you are a developer, you have probably used `print` statements to debug your Python application. When you are trying to obtain information for debugging your application, you need much more information than you might think, such as timestamps, related modules, and types. You can use the `print` command in a small application, but this strategy quickly becomes unwieldy in a large, complex application in which multiple modules are communicating and sharing data with each other.

To resolve this issue, you need a well-built **logging** module that can write all kinds of information related to your application to one of your output streams (e.g., a **console** or **log file**) in a structured and predictable manner.

## Levels of Logging

In the Python logging module, there are five standard levels related to the **severity** of the events:

- Debug(10): Used mostly in non-production environments to diagnose issues or get specific types of information.
- Info(20): Used to output event information and create a trace of execution.
- Warning(30): Used when something unexpected happens that may cause problems in the future.
- Error(40): Used when a problem occurs that disrupts the normal functioning of the application.
- Critical(50): Used when one or more parts of the application fail to run.

Based on the severity of the event, you can create a log or even config the logger to only log events either below or above a particular severity level.

In the later sections of this article, you'll learn how to use these severity levels in the logging module.

## Using Logging Module in Python

To understand the concept of logging in Python, create a project directory (`python-logging`) by running the following commands in the terminal:

```bash
mdkir python-logging
cd python-logging
```

In the `python-logging` directory, create a `main.py` file and add the following code to it:

```python
# 1
import logging

# 2
logging.debug('This is a debug message')
logging.info('This is a info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
```

In the above code, you are performing the following:

1. You are importing Python's inbuilt `logging` module.
2. You are using the `logging` module helper functions to print the logs related to five severity levels on the console window.

Run the `main.py` file by running the following command in your terminal:

```bash
python main.py
```

If you see the output in the terminal window, you'll see that logs related to only warning, error, and critical severity level were printed.

![Default logs created by the logging module.](https://www.honeybadger.io/images/blog/posts/python-logging/default_logs_logging_module.png)

This is because, by default, the logging module only prints logs with a severity level equal to or greater than that of the warning.

## Configuring the Severity Level

To override the default severity level, you can configure the logging module by using the `basicConfig` function and passing the `level` parameter to it. To do so, update your existing `main.py` file to include the following:

```python
import logging

logging.basicConfig(level=logging.DEBUG) # added

logging.debug('This is a debug message')
logging.info('This is a info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
```

In the above code, you are performing the following:

1. You are using the `basicConfig` function to configure the `logging` module's settings. You have specified the `level` parameter as `logging.DEBUG`, which configures the `logging` module to log all messages that have a severity level equal to or greater than the debug level.

Run the `main.py` file by running the following command in the terminal:

```bash
python main.py
```

This time, you'll see all the logs printed on your terminal window.

![Logs of all severity levels created by the logging module.](https://www.honeybadger.io/images/blog/posts/python-logging/logs_severity_levels.png)

Logs of all severity levels created by the Logging module.

Based on your requirements, you can configure the severity level of your application. For example, while developing the application, you can set the `level` parameter to `logging.DEBUG`, and in a production environment, you can set it to `logging.INFO`.

## Writing Logs to a Log File in Python

When you are developing an application, it is convenient to write the logs to the console. However, in a non-development environment, you might need a way to store the application logs persistently. For this purpose, you can store your logs in a **log file**. This log file can be accessed at any time and used to debug issues or extract all kinds of information related to the events happening in your Python application.

To create a log file, you can use `filename` and `filemode` parameters in the `basicConfig` function.

Replace the contents of the `main.py` file with the following code:

```python
import logging

# 1
logging.basicConfig(filename='app.log', filemode='w', level=logging.DEBUG)

logging.warning('This is a warning message')
logging.debug('This is a debug message')
```

In the above code, you are performing the following:

1. You are specifying the `filename` and `filemode` parameters in the `basicConfig` function. This configuration creates an `app.log` file at the root of your Python application and stores all the logs in it rather than printing them to the console window. The `filemode` parameter should be in either **write** (`w`) or **append** (`a`) mode to give permission to Python to create and write log files. An important thing to note here is that the write mode will recreate the log file every time the application is run, and append mode will add the logs to the end of the log file. The append mode is the default mode.

Run the `main.py` file by running the following command in the terminal:

```bash
python main.py
```

Next, check for the `app.log` file at the root of your application. You'll see the following log messages:

![Logs stored in a log file.](https://www.honeybadger.io/images/blog/posts/python-logging/file_logs.png)

Logs are stored in a log file.

## Customizing the Log Message Format

You can log more information about the events happening in your Python application by using the `format` parameter in the `basicConfig` function. The `format` parameter allows you to customize your log message format and include more information, if required, with the help of a [huge list of attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes). For example, you can include a **timestamp** in your log message.

To do so, update the `main.py` file by replacing its content with the following code:

```python
import logging

# 1
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s')

logging.info('This is a custom format log')
```

In the above code, keep the following in mind:

1. The `format` parameter takes a string in which you can use LogRecord attributes in any arrangement you like. The `%(asctime)s` attribute adds the current timestamp to the log message.

Run the above code by executing the following command in the terminal:

```bash
python main.py
```

In the terminal window, you'll see the log message with the timestamp information and a custom format:

![Logs with a custom format.](https://www.honeybadger.io/images/blog/posts/python-logging/logs_custom_format.png)

## A Detailed Example

Now that you have an idea of how logging works in Python, let's extend the examples to an actual Python program and see how logging fits in. In this example, you'll see how to use the `logging.error` function in the `try-except` block.

In the `main.py` file, replace the existing code with the following:

```python
import logging

logging.basicConfig(
    format='%(asctime)s - %(levelname)s - %(name)s - %(message)s')

# 1
a = 10
b = 'hello world'

# 2
try:
    c = a / b
# 3
except Exception as e:
    logging.error(e)
```

In the above code, you are performing the following:

1. You are declaring two variables, an *int* `a` and a *string* `b`.
2. In the `try` block of the `try-expect` statement, you are performing a division operation between an *int* (`a`) and a *string* (`b`). As you probably already know, this is not a valid operation and will cause an exception.
3. In the `expect` block, you are catching the exception (`e`) and logging it using the `logging.error` function.

Run the above code by executing the following command in the terminal:

```bash
python main.py
```

In the terminal window, you'll see a custom-formatted log message stating the unsupported operation error:

![Log generated due to an exception.](https://www.honeybadger.io/images/blog/posts/python-logging/exception_logs.png)

## Third-Party Python Modules for Logging

Besides Python's inbuilt logging library, there are also third-party logging libraries that you can use in Python applications.

### Logging in **Django**

[Django](https://www.djangoproject.com/) is the most popular Python-based Web application development framework. Internally, Django uses the Python logging module but allows the developer to customize the log settings by configuring the `LOGGING` config variable in the `settings.py` file.

For example, the following configuration writes all log output to the `debug.log` file at the root of the application:

```python
LOGGING = {
    'version': 1,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': 'debug.log'
        }
    },
    'loggers': {
        'django': {
            'handlers': [
                'file'
            ],
            'level': 'DEBUG'
        }
    }
}
```

This setting updates the logs generated by the Django Root Module, Django Server, Django Web Requests, and Django database queries. For more information on configuring logging in Django, check out  [Django’s documentation](https://docs.djangoproject.com/en/3.2/topics/logging/#configuring-logging).

### Logging in Flask

[Flask](https://flask.palletsprojects.com/), like Django, is also a Python-based Web application development framework, but it is smaller in size than Django. It also uses Python's built-in logging modules and provides an `app.logger` function to configure log settings.

For example, to send logs to the console in the JSON format, you can use the following configuration in a Flask app before initializing the `app`:

```python
from logging.config import dictConfig

dictConfig({
    'version': 1,
    'formatters': {
        'json': {
            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter'
        }
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'json'
        },
    },
    'root': {
        'level': 'INFO',
        'handlers': ['console']
    }
})

app = Flask(__name__)
```

For more information regarding logging in Flask, you can check out [Flask’s logging documentation](https://flask.palletsprojects.com/en/2.0.x/logging/).

### Storing Logs in JSON Format

Instead of logging the events in a plain text format, you might want to output them as **JSON** objects. With the JSON format, you can easily add records to a database or query them using the JSON-based query language. For this use case, you can utilize the [python-json-logger](https://pypi.org/project/python-json-logger/) library to store logs in JSON format.

To install this package in your Python application, run the following command in the terminal:

```bash
pip install python-json-logger
```

The basic code to start using this library is as follows:

```python
import logging
import json_log_formatter

formatter = json_log_formatter.JSONFormatter()

json_handler = logging.FileHandler(filename='log.json')
json_handler.setFormatter(formatter)

logger = logging.getLogger('my_json')
logger.addHandler(json_handler)

logger.error('An error occurred', extra={'type':'fatal'})
```

This package creates JSON logs, as shown in the following image:

![Logs in JSON format.](https://www.honeybadger.io/images/blog/posts/python-logging/logs_json.png)

## Conclusion

It is very important to monitor events occurring in an application, and one of the most recommended ways is to use logging. Without keeping track of what's happening in your application and how the users are utilizing it, you will never be able to identify performance bottlenecks. There is a saying that *you can only improve what you can measure*. If you are unable to improve your system, you will lose customers over time. Therefore, it is recommended to use logging in your applications.

If you are looking for a robust, cloud-based system for real-time monitoring, error tracking, and exception-catching, you might love [Honeybadger](https://www.honeybadger.io/). You can use it with any framework or language, including Python, Ruby, JavaScript, and PHP.