Writing Custom Log Handlers In Python
Tags: python, logs, microservices, devops
Published: 2022-11-02 13:50:26

Have you ever wondered how logging services like logz .io or loggly .com ship your logs, onto their platform? For example logz .io has an [opensource](https://github.com/logzio/logzio-python-handler) logging handler which they use to transport logs from your application to their platform where further analysis are carried out on these logs to enable you derive substantial insights on your application's performance. In this short article we are briefly going to cover the basics of writting user-defined log handlers using the python [standard logging](https://docs.python.org/3/library/logging.html) package. 

You might be wondering why i chose to write about this rather "odd" topic, but the importance of a good logging system cannot be overstated, whether your application is a web application, mobile  or even an operating system, logging enables you to derive useful informations on how your application is performing. A good logging system helps in catching errors as they happen, instead of relying on users to report them, the majority of whom simply wouldn’t or wouldn’t know how to do it. 

## The logging.StreamHandler
The python standard logging package provides a **Handler** class which basically defines how a particular log message is handled, we can configure a simple handler that prints log messages to the terminal like this: 

```
LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}
```
We can then initialize this configuration, in a log.py _assuming the log.py file is the application we want to log_

```
import logging
from logging.config import dictConfig


dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()
```
When we run the log.py file `python3 log.py`; we can see a similar output to this shown on the terminal

![Logging to the console with python](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gh7ivkjvue766eejd0hp.png)
As you can see we've been able to configure a handler that logs directly to the console(stdout), keep in mind you don't need to create any congifuration to log to the terminal, since the StreamHandler is the default handler for the logging package. The logging package also comes with an array of already made **Handlers** you can use asides the StreamHandler, some of which are listed below:

- **FileHandler**: which sends logging output to a disk file
- **NullHandler**: It is essentially a ‘no-op’ handler for use by library developers. Which i've never had any reason to use :)
- **BaseRotatingHandler**: which is basically another type of FileHandler for special use cases
- **SocketHandler**: which sends logging output to a network socket.
- **SysLogHandler**: which sends logging messages to a remote or local Unix syslog.
- **SMTPHandler**: which sends log messages to an email address, using SMTP.
- **HTTPHandler**: which supports sending logging messages to a web server, using either GET or POST semantics. 

There are several handlers provided by the logging package that can be found [here](https://docs.python.org/3/library/logging.handlers.html), some of which you might never have a use case for since they are built for very specific use cases.

## Custom Handlers 
Supposing we are building a cloud based logging as a service software like [logz.io](https://logz.io), where we'd like to store, retrieve and analyze our logs for better insights; python provides a low-level "api" to enable us transports logs from python applications easily, using the `logging.Handler` class. In the following example we are going to create a simple Custom Handler that writes to a ".txt" file. Obviously log messages are not written to txt files but this is to illustrate how to write Custom Log Handlers in a simple way. 

```
class SpecialHandler(logging.Handler):

    def __init__(self )-> None:
        self.sender = LogSender()
        logging.Handler.__init__(self=self)

    def emit(self, record) -> None:
        self.sender.writeLog(record)
```
In the code above we declare a class named `SpecialHandler`, which inherits from the `logging.Handler` base class, this basically allows us to extend the base Handler so as to make it fit our use. We then declare the `__init__` method and initiates the class which handles each log record as it is retrieved i.e `LogSender`, the `emit(self, record)` method is quite special as it responsible for sending each log record to the specified reciever. 

```
 class LogSender:
    def __init__(self) -> None:
        pass
    
    def writeLog(self, msg: logging.LogRecord) -> None:
        with open("application.txt",'a',encoding = 'utf-8') as f:
            f.write(f"{msg} \n")
```
The `LogSender` class acts as the retriever of log records as they are sent from the `emit` method of the Handler's class; the **LogSender** class declares a `writeLog` method, which accepts any argument of type `msg: logging.LogRecord`. On reception of a record it opens a txt file `application.txt` in append mode and writes the log message to the file, adding a new line at the end of it.
This is basically all it takes to create a customHandler, we can configure our `LOGGING Dictionary` to recognize this new handler with a few lines of code, like so:
```
LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}
```
Keep in mind, the writer.SpecialHandler is our custom handler class written in a writer.py file. To initialize this changes we can import the writer.py file into our log.py file and load the logging configuration into dictConfig again, but this time with a custom handler that writes to our application.txt file. 

```
import logging
from logging.config import dictConfig
import writer #writer.py contains our SpecialHandler and LogSender class respectively

LOGGING = {
    "version": 1, 
    "disable_existing_loggers": False,
    "formatters": {
        "stdformatter": {"format": "DateTime=%(asctime)s loglevel=%(levelname)-6s  %(funcName)s() L%(lineno)-4d %(message)s call_trace=%(pathname)s L%(lineno)-4d"},
    },
    "handlers": {
        "stdhandler": {
            "class": "logging.StreamHandler",
            "formatter": "stdformatter",
            'stream': 'ext://sys.stdout'
        },
        "specialhandler":{
            "class" : "writer.SpecialHandler",
            "formatter": "stdformatter",
        }
    },
    "loggers" : {
        "root": {
            "handlers": ["stdhandler", "specialhandler"], 
            "level": "DEBUG",
            "propagate": True
            }
        }
}

dictConfig(LOGGING)
logger = logging.getLogger()
logger.debug("logger started")

def logToConsole():
    logger.info("Application started, succesfully")
    logger.info("Logging to the console")
    logger.info("Finished logging to console!")

# call the function
logToConsole()
```
When we run `python3 log.py` we can see an application.txt file is created within the current directory, containing all the log records. 

![custom handler output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gurb2uxxd5v4pw50xccn.png)

## Conclusion
Logging applications can save us hours in debugging hidden bugs within our application, as well as enabling to derive useful insights about the performance, operatability and needs of applications without any hassles. In this short article, we've seen how to write basic custom log handlers for our applications, extending to fit our specific needs. 

Let me know what you think about this article and possibly any amendments that can be made to improve it's user experience, thanks for reading and have a great time!