Get Started Quickly With Python Logging
Tags: logging, programming, coding, devops
Published: 2019-08-13 13:28:17

Picking up from the previous logging articles on how to get <a href="https://www.scalyr.com/blog/get-started-quickly-csharp-logging/">started logging with C#</a> and <a href="https://www.scalyr.com/blog/get-started-quickly-java-logging/">Java</a>,&nbsp;today we'll be looking at how to get up and running quickly with Python logging.

Even if you've already read the previous articles, this post is worth a read. It will cover new ground, like the basics around application logging in Python and a few other things, such as
<ul>
 	<li>Configuring the logging module.</li>
 	<li>What to log and why.</li>
 	<li>The security implications of logging.</li>
</ul>
So what are you waiting for? Keep reading, and let's get a simple project set up to begin working with.

<img class="aligncenter wp-image-735" src="https://library.scalyr.com/2018/07/19175338/Python-Scalyr-Colors-with-Log.png" alt="" width="301" height="295"><!--more-->
<h2>The Simplest Python Logging That Could Work</h2>
For this demo, I'm going to use the free community edition of <a href="https://www.jetbrains.com/pycharm/">JetBrain's PyCharm IDE</a>. If you haven't used it before, go ahead and <a href="https://www.jetbrains.com/pycharm/download/">download</a> it and get it installed. Once you have PyCharm installed, open it up.

At the "Welcome" screen, click on the "Create New Project."

<img class="alignnone wp-image-1573" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-create-new-project.png" alt="" width="558" height="346">

<img class="aligncenter size-large wp-image-2481" src="https://library.scalyr.com/2018/02/30185233/pycharm-create-new-project-1024x635.png" alt="" width="1024" height="635">

On the next screen, we'll need to set a few options. First, set the location. It should default to the PyCharm project location. You can call the project whatever you want. I'm calling mine "quick-python-logging." Expand the section below the "Location" option, and you'll see options related to the Python virtual environment and interpreter. If this isn't set up for Python3, refer to this <a href="https://www.jetbrains.com/help/pycharm-edu/configuring-python-interpreter.html">guide</a>, which will get you up and running.

Once you have all of those options in place, go ahead and click "Create."

<img class="alignnone wp-image-1580" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-project-setup.png" alt="" width="562" height="349"><img class="aligncenter size-large wp-image-2482" src="https://library.scalyr.com/2018/02/30185303/pycharm-project-setup-1024x635.png" alt="" width="1024" height="635">

PyCharm will create a new virtual environment for us, and after a minute or two, we'll have an empty Python project ready for us to start writing code. Let's create a new Python file by right-clicking on the project name and selecting "New Python File" from the "New" submenu dropdown.

<img class="alignnone wp-image-1578" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-empty-project.png" alt="" width="567" height="331"><img class="aligncenter size-large wp-image-2483" src="https://library.scalyr.com/2018/02/30185324/pycharm-empty-project-1024x597.png" alt="" width="1024" height="597">

Call it main.py.

<img class="alignnone wp-image-1579" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-new-python-file.png" alt="" width="564" height="215">

<img class="aligncenter size-full wp-image-2484" src="https://library.scalyr.com/2018/02/30185339/pycharm-new-python-file.png" alt="" width="698" height="266">

Go ahead and update our new main.py file with the following code:
<pre class="lang:python decode:true">if __name__ == "__main__":
    print("Hello, Python!")

</pre>
Now, let's just verify that your interpreter is set up correctly. Go to&nbsp;<code>Run</code>&nbsp;&gt;&nbsp;<code>Run 'Main'</code>&nbsp;or use the shortcut&nbsp;<code>Shift</code>&nbsp;+&nbsp;<code>F10</code>&nbsp;(Windows and Linux) or&nbsp;<code>Control</code>&nbsp;+&nbsp;<code>R</code>&nbsp;(Mac). If everything is set up properly, PyCharm will bring up the run output window, where we'll see our message. It should look like this:

<img class="alignnone wp-image-1572" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-run.png" alt="" width="568" height="105"><img class="aligncenter size-large wp-image-2485" src="https://library.scalyr.com/2018/02/30185400/pycharm-run-1024x189.png" alt="" width="1024" height="189">

While we can use the print statement to display messages to the console---and who hasn't done this to debug an issue?---this isn't an ideal way for us to handle logging in any application. Application logging is more than just printing messages to the screen.
<h2>What Is Application Logging?</h2>
So what exactly is application logging? Well, if you happened to have read the <a href="https://www.scalyr.com/blog/get-started-quickly-csharp-logging/">C# logging post</a>, you'll know that the definition of application logging given in that article is
<blockquote>Application logging involves recording information about your application’s runtime behavior to a more persistent medium.</blockquote>
There are a lot of pieces of information that might appear in a log statement. Applications generally have a lot in common with what we'll log. There's also additional information that your application may need to log around business processes, and that will be useful to consider.

Conceptually, we want to log anything that can give us insight into the behavior of the application while it's running. We'll also persist these messages for later review and analysis.
<h2>What's the Motivation For Logging? Why Bother?</h2>
Imagine this all-too-common scenario. You recently finished a major upgrade to an application, and it was just released to production. All the production tests have passed. The weekend is here, so it's time to sit back and relax---or as it turns out, not exactly.

<img class="size-full wp-image-2486 alignleft" src="https://library.scalyr.com/2018/02/30185449/Is-there-some-unknown-factor-that-might-cause-this-application-to-fail.png" alt="" width="300" height="300">Reports start coming in that something isn't working, so you log in and take a look at the logs...except you don't see anything going wrong. No exceptions appear, and no other indication is given that the application isn't behaving properly. It was on the backlog to overhaul the logging around a few of the business processes, but there wasn't time in the last quarterly release to work on it.

See, in many of these situations, the organizations didn't have proper instrumentation or logging. Is the application really OK? It passed through the quality assurance acceptance tests, but are&nbsp;<em>those</em> OK? Are they good enough? Is there some unknown factor that might cause this application to fail spectacularly at a pivotal moment in production?

Properly logging your application can answer all of these questions and more. Logging can provide us with information on the health of an application, and it can allow us to
<ul>
 	<li>Reconstruct what went wrong with a business process.</li>
 	<li>Review requests and contextual information in the case of a security breach.</li>
 	<li>Provide data for marketing analytics.</li>
 	<li>Monitor for policy violations and regulations.</li>
</ul>
You need a logging strategy in your application to ensure that the proper data is being logged and managed.
<h2>What You Should (and Shouldn't!) Log</h2>
There are a lot of things we can put into a log file, but there are some pieces of information that we should never, ever include in log output. You shouldn't log any form of sensitive business data or <a href="https://en.wikipedia.org/wiki/Personally_identifiable_information">personally identifiable information (PII)</a>.

Sensitive business data is a bit generic. What is and isn't sensitive can vary depending on the industry, but at the very least, you&nbsp;shouldn't log the following:
<ul>
 	<li>Account numbers</li>
 	<li>Credit card numbers (including expiration dates and CVV codes)</li>
 	<li>Passwords</li>
 	<li>Session identifiers</li>
 	<li>Encryption keys</li>
</ul>
Personally&nbsp;identifiable information includes anything that can be used to identify an individual. This list is not exhaustive, but it would include such things as
<ul>
 	<li>Social security numbers</li>
 	<li>Birth dates</li>
 	<li>Full names</li>
 	<li>Email addresses</li>
 	<li>Driver's license numbers</li>
</ul>
In short, treat your logs as if they'll one day become public. Log data is a great source of information for an attacker to exploit.&nbsp;The <a href="https://www.owasp.org/index.php/Main_Page">Open Web Application Security Project (OWASP)</a> has a great <a href="https://www.owasp.org/index.php/Logging_Cheat_Sheet">cheat sheet on logging</a> that would definitely be worthwhile to read over for more information.
<h2>Enter the Python Logging Framework</h2>
Let's get back to some code. It isn't a secret that Python has a pretty extensive standard library. The Python <a href="https://docs.python.org/3/library/logging.html">logging</a> module is a very capable logging framework, so let's integrate that into our application.

Go ahead and modify your main.py to import the logging library and add a couple of log messages to our application.
<pre class="lang:default decode:true">import logging

if __name__ == "__main__":
    logging.warning("I'm a warning!")
    logging.info("Hello, Python!")
    logging.debug("I'm a debug message!")</pre>
Now, if you run this, you might see something unexpected.

<img class="alignnone wp-image-1577" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-demo-log-warning.png" alt="" width="514" height="97"><img class="aligncenter size-large wp-image-2487" src="https://library.scalyr.com/2018/02/30185514/pycharm-demo-log-warning-1024x193.png" alt="" width="1024" height="193">

Where are our other messages? Why was only the warning message printed? Well, it has to do with <a href="https://www.scalyr.com/blog/logging-levels/">logging levels</a>. We'll get to handling these in a moment, but let's first look at a couple of other things regarding our logging code.

While we can continue to use our logging library to print messages to the console, this isn't probably going to work long-term. We know that we need to persist the information, so why not a file? This is straightforward to do, but we need to look at how to configure the logging module.
<h2>Configuring the Logger</h2>
The Python logging module is broken into four different abstractions: <em>loggers</em>, <em>handlers</em>, <em>filters</em>, and <em>formatters</em>.

<em>Loggers</em> expose the primary interface that we use to log events from the application. <em>Handlers</em>, well, handle where the log event is sent. Using different handlers, we can have log events go to different files or some to one file and the rest to syslog. We can change what log events get written or add contextual information to an event through using a <em>filter</em>. Finally, log <em>formatters</em> allow us to specify the layout of our data when it gets written.

We can also configure logging at different module and class levels. This allows us to have finer-grained control over what's being logged and where it's being sent.
<h2>Editing the Code</h2>
Let's get our current logging demo writing to a log file. We'll use the basicConfig method on the logging module to quickly configure logging for our application. Open up your main.py and change it to the following:
<pre class="lang:python mark:3 decode:true ">import logging

logging.basicConfig(filename='demo.log')

if __name__ == "__main__":
    logging.warning("I'm a warning!")
    logging.info("Hello, Python!")
    logging.debug("I'm a debug message!")</pre>
Run this and you'll see that a file named demo.log is created in your project directory. Open that up, and you'll find that it contains the following:

<img class="aligncenter size-full wp-image-2488" src="https://library.scalyr.com/2018/02/30185534/pycharm-demo-log.png" alt="" width="626" height="149"><img class="alignnone wp-image-1574" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-demo-log.png" alt="" width="525" height="125">

Before we move on, let's get the rest of our messages writing to the log file. Earlier, I mentioned our other messages weren't appearing due to the logging level. Let's change that so we're logging everything.

Go to your main.py and update the configuration:
<pre class="">import logging

logging.basicConfig(filename='demo.log', level=logging.DEBUG)

if __name__ == "__main__":
    logging.warning("I'm a warning!")
    logging.info("Hello, Python!")
    logging.debug("I'm a debug message!")</pre>
Now run the application. You'll see the rest of our messages in the file.

<img class="alignnone wp-image-1575" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-demo-log-all.png" alt="" width="518" height="111"><img class="aligncenter size-full wp-image-2489" src="https://library.scalyr.com/2018/02/30185614/pycharm-demo-log-all.png" alt="" width="690" height="148">

So, what did we do? Logging frameworks break down the type of messages being logged into different categories or severity levels. The actual breakdown may be different based on the logging framework. Python uses the following priority levels, starting with the highest priority first:
<ul>
 	<li>Critical</li>
 	<li>Error</li>
 	<li>Warning</li>
 	<li>Info</li>
 	<li>Debug</li>
 	<li>Notset</li>
</ul>
By default, the Python logging module is set at the warning logging level. This means that any requests at the warning level or higher priority are logged to the destination. The rest are ignored. This is why our info and debug messages weren't showing up in the log.
<h2>Formatting Our Messages</h2>
Now that we have our log messages being written to a persisted medium, we need to make our messages more useful. Let's configure a formatter to create a better layout and to include additional information for helping with debugging our application.

Add the following to your main.py:
<pre class="lang:default mark:3-5 decode:true ">import logging

logging.basicConfig(filename='demo.log',
                    level=logging.DEBUG,
                    format='%(asctime)s - %(name)s - %(threadName)s -  %(levelname)s - %(message)s')

if __name__ == "__main__":
    logging.warning("I'm a warning!")   
    logging.info("Hello, Python!")
    logging.debug("I'm a debug message!")</pre>
Now run the application again and open up the demo.log file.

<img class="aligncenter size-large wp-image-2490" src="https://library.scalyr.com/2018/02/30185637/pycharm-demo-log-format-1024x136.png" alt="" width="1024" height="136"><img class="alignnone wp-image-1576" src="https://staging-design.scalyr.com/wp-content/uploads/2018/07/pycharm-demo-log-format.png" alt="" width="519" height="69">

Now we're getting somewhere. Our little logger is much more useful. So what's going on with this format statement? Here's the info message from our log file:
<pre class="lang:default decode:true">2018-01-27 12:54:48,312 - root - MainThread - INFO - Hello, Python!</pre>
Let's break it down.
<blockquote>2018-01-27 12:54:48,312</blockquote>
First thing we log is the time stamp through the Python asctime variable. This is an ISO8601-compliant time stamp.
<blockquote>root</blockquote>
This is the name of the logger. You can view loggers as a tree, and since we didn't give our logger a name, we're given the root logger by default.
<blockquote>MainThread</blockquote>
MainThread is the name of the current thread. This is very useful when logging in a multi-threaded application.
<blockquote>INFO</blockquote>
This is the logging level of the message.
<blockquote>Hello, Python!</blockquote>
Finally, we have the message itself.
<h2>Now What?</h2>
Logging frameworks are a powerful and necessary tool. There's a lot more we can do with our logging module, such as configuring logging for multiple modules or using a configuration file. We also need to decide on an overall<a href="https://www.scalyr.com/blog/application-logging-practices-adopt/"> logging strategy</a>.

<img class=" wp-image-2491 alignright" src="https://library.scalyr.com/2018/02/30185710/Logging-frameworks-are-a-powerful-and-necessary-tool-1.png" alt="" width="292" height="292">There are far more options that can be used to configure the logger, handlers, and formatters. I encourage you to experiment and see what works best for your particular application. The Python documentation has a lot of information on the <a href="https://docs.python.org/3/library/logging.html#module-logging">logging module</a>, as well as a <a href="https://docs.python.org/3/howto/logging.html#logging-basic-tutorial">basic</a> and <a href="https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial">advanced</a> logging guide.

Stay tuned to the blog here at Scalyr for more articles on logging and logging best practices. Scalyr makes&nbsp;<a href="https://scalyr.com/blog/what-is-log-aggregation-and-how-does-it-help-you/">a log aggregation tool</a>,&nbsp;meaning that once you have lots of data spread out across many files, they’ll help you put them all in one place, search them, and visualize the data in them.

For now, take a look at the applications you're working on and consider the current logging strategy in use. There's most likely room for improvement, so what are you waiting for?

<em>This post was written by <a href="http://www.caseydunham.com/">Casey Dunham</a>.&nbsp; Casey, who recently launched his own security business, is known for his unique approaches to all areas of application security, stemming from his 10+ year career as a professional software developer. His strengths include secure SDLC development consulting; threat modeling; developer training; and auditing web, mobile, and desktop applications for security flaws.</em>