Log rotation with Python
Tags: python, logging, beginners
Published: 2023-10-25 18:11:21

In this post, I show how to implement [log rotation](https://en.wikipedia.org/wiki/Log_rotation) in a Python project.

## What's log rotation?

From [Wikipedia](https://en.wikipedia.org/wiki/Log_rotation): 

> 
In information technology, log rotation is an automated process used in system administration in which log files are compressed, moved (archived), renamed or deleted once they are too old or too big (there can be other metrics that can apply here). New incoming log data is directed into a new, fresh file (at the same location)

So, it's important to consider log rotation specially in that systems that produce a great quantity of log messages.

There are different criteria of log rotation and in this post I'll focus on timed log rotation and file size rotation.

## How we implement logging in Python?

Super easy, just using the **logging module**.

The logging module offers a default logger named "root". This logger allows you to get started with no configuration. 

```
import logging
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
```

This module permit a lot of customizations like:
- Logging severity level
- File name
- Log format
- etc.

You can create a custom logger using the class **Logger,** defining a **basic configuration** (_date format_, _log format_, etc.).

```
  logging.basicConfig(
        format='%(asctime)s %(levelname)s %(message)s',
        level=logging.DEBUG,
        datefmt='%m/%d/%Y %I:%M:%S %p',
    )

    # Log formatter definition
    log_formatter = logging.Formatter(fmt=' %(asctime)s %(levelname)s %(message)s')

    # Getting the logger
    logger = logging.getLogger('my-logger')
    
    # Using the logger
    logging.debug('This is a debug message')
    logging.info('This is an info message')
    logging.warning('This is a warning message')
    logging.error('This is an error message')
    logging.critical('This is a critical message')
```
To implement different ways to log information, Python uses **log handlers**.

The **log handlers** are components that effectively writes logs, and they give you the opportunity of:
- Display logs in the console (StreamHandler)
- Write logs on files (FileHandler)
- Write logs on files with log rotation based on file size criteria (RotatingFileHandler)
- Write logs on files with log rotation based on time conditions (TimedRotatingFileHandler)
- Sending logs via SMTP (SMTPHandler)
- etc.

You can create your custom handler or extend a pre-existing handler. 

## File Logging with Python

To implement file logging, we'll use the **FileHandler**, a handler able to write logs to the file system. 
Below is shown how to define this handler using the same basic configuration defined previously.

```
    # File handler definition
    fh: logging.FileHandler = logging.FileHandler(
        filename=f'./logs/my-app-file.log',
        mode='a'
    )

    # Setting the formatter for the handler
    fh.setFormatter(log_formatter)

    # Setting the severity level for the handler
    fh.setLevel(logging.INFO)

    # Adding handler to the logger
    logger.addHandler(fh)

    #Trying the handler using the handler
    logger.info('This log is an info message')
```
This handler simply writes log messages to a file named my-app-file.log.


## Log rotation with Python

### File size log rotation

To implement logging rotation based on the size of the file, we'll use the **RotatingFileHandler**. This handler is used to write logs on the file system and to trigger a rotation when a specific size of the file is reached. 
Below is shown how to define this handler using the same basic configuration defined previously.

```
    # Rotation file handler definition
    rfh: RotatingFileHandler = RotatingFileHandler (
        filename=f'./logs/my-app.log',
        maxBytes=100,
        backupCount=5
    )

    # Setting the formatter for the handler
    rfh.setFormatter(log_formatter)

    # Setting the severity level for the handler
    rfh.setLevel(logging.INFO)
    
    # Adding handler to the logger    
    logger.addHandler(rfh)

    # Trying the handler using the handler
    logger.info('This log is an info message')
```

In this example, we've set a max dimension of 100 bytes and a backup count of 5 log files (this means that we keep at most 5 files from the log history).

A result of this rotation after writing more than 1024 bytes could be something like shown below:

```
my-app.log
my-app.log.1
my-app.log.2
my-app.log.3
my-app.log.4
my-app.log.5
```

### Timed log rotation with Python

To implement file rotation logging based on time event, we'll use the **TimedRotatingFileHandler** that is the handler used to write logs on the file system and trigger a rotation when a time condition is satisfied.
Below is shown how to define this handler using the same basic configuration of the previous examples.

```
    rth: TimedRotatingFileHandler = TimedRotatingFileHandler(
        filename=f'./logs/{log_file_prefix}.log',
        backupCount=5,
        when='midnight',
    )
    rth.setFormatter(log_formatter)
    rth.setLevel(logging.INFO)
    rth.namer = lambda name: name.replace(".log", "") + ".log"


    logger.addHandler(rth)
    logger.info('This log is an info message')
```

In this example, we've set a daily rotation triggered each midnight and a backup count of 5 files. 

A result of this rotation after 5 days could be something like shown below:

```
my-app.log
my-app.2023-10-12.log
my-app.2023-10-11.log
my-app.2023-10-10.log
my-app.2023-10-09.log
my-app.2023-10-08.log
```

For this handler, we've defined a _namer_ with a lambda function that has the goal to preserve the extension of the log files. This _namer_ get in on the action when a rotation happens.

#### Custom File timed log rotation with Python

It's possible to create the own log handler extending a pre-existing handler. For instance, we can extend the **TimedRotatingFileHandler** class in a way to introduce, for each log message emission, a check about the presence of the file. In this way, if someone or something deletes the file, the handler is able to create it again, and we don't miss further log messages.

Let's create the class:

```
class TimedRotatingFileHandlerWithFileCheck(TimedRotatingFileHandler):
    """
    Handler for logging to a file, rotating the log file at certain timed
    intervals. This handler based on TimedRotatingFileHandler offers a file 
    presence check for each log emission.
    """
    def __init__(self, filename, when='h', interval=1, backupCount=0,
                 encoding=None, delay=False, utc=False, atTime=None,
                 errors=None):
        super().__init__(filename, when, interval, backupCount, encoding, delay, utc, atTime, errors)

    def emit(self, record):
        if not os.path.isfile(self.baseFilename):
            os.mknod(self.baseFilename)
        super(TimedRotatingFileHandlerWithFileCheck, self).emit(record)
```

In this version, before to emit a log, we check the presence of the file.  

## Conclusions

In Python, logging is super easy! The **logging module** offers a lot of possibilities to customize and implement the strategy more suitable for you. The presence of a lot of handlers and the possibility to have a high level of customization makes this module very useful.
In my opinion, having a good and balanced logging system is a must for each system or application.

Suggestions and corrections are welcome.

## Credits
- [Abhinav Ajitsaria from realpython.com](https://realpython.com/python-logging/)
- [Son Nguyen Kim from toptal.com](https://www.toptal.com/python/in-depth-python-logging)