Advanced Logging With Python
Advanced logging features in Python using the standard logging module
I work as a software developer for a company that develops climate control systems for greenhouses. These climate control systems control various aspects of a greenhouse to create an optimal climate for growing crops. We connect the system to several sensors, and the system sends the measurements of these sensors to a central location in the cloud.
The grower, our customer, can access these measurements through various dashboards that include graphs and reports. Last week, we received a customer call through the helpdesk that he did not see any new measurements come in. Other customers had no problems; it looked like a problem at his greenhouse.
To determine why the application stopped sending measurements, I logged in to the local system. I changed the default logging level from Warning to Information in the logging configuration file. After the change, an information logging message informed me that the application could not connect to the internet.
After asking the customer if something changed in the last few days about the network infrastructure, we found out that a system administrator installed a new firewall. This firewall did not allow our system to connect to the Internet. Soon after speaking with the system administrator, he resolved the issue.
I use logging in all the applications I develop. This helps me see the flow of an application at run-time and see where a problem or error occurs.
Before, I have written an article about the basics of logging in Python. In this article, I will look at the more advanced features of the standard Python logging module, such as configuring and structuring.
Configuring your logger
By configuring, I mean setting the logging level and the destination of the logging. There are two significant ways to configure your logger. You can use a configuration file, which I think is the best way for configuration, and you can configure via code.
You can use two types of configuration files, the standard configuration file format, and a JSON file that follows the Dictionary Schema. Let’s start with the first one, the oldest, the configuration file format.
Configuration via a configuration file
The Python team based the fileconfig() format on the functionality of configparser. See below a configuration file using this format.
Each logging configuration must contain three sections called “loggers”, “handlers”, and “formatters”. Depending on what you define in these required sections, other sections are needed.
In the configuration file below, we defined two loggers, the root logger and a custom one, the “main” logger. We also defined two handlers, a console handler, and a file handler. For each of the keys we described in the sections we also have to create a key section.
So as we defined, keys=console, file in the handlers sections, we also have to define a [handler_console] and a [handler_file] section. The same goes for the formatter.
In the [logger_root] section, we see that the root logger uses the DEBUG level and that it reports the logging via the console handler.
In the [handler_file] section, we define the characteristics of the file handler. We want to use the FileHandler so that we send the logging to a file. The handler uses the standard formatter, which we define in the [formatter_standard] section. Through the use of args on row 29, we can specify the filename.
To use this logging configuration, we can use the following Python code. I stored the configuration from above in a file called log.config and read it via the logging.config.fileconfig('log.config') statement.
The new version of the configuration API uses a dictionary. This version can be called by using dictConfig(). The dictionary API offers additional functionality such as the configuration of filter objects. Therefore, dictConfig() is preferred.
Configuration via a dictionary configuration file
The dictConfig() format is described in the Configuration dictionary schema. The options are more or less the same as with the fileConfig. See the configuration below, it defines the same configuration as before but with the dictionary format.
The argument of the dictConfig() method is the actual dictionary containing the configuration. It does not as with fileConfig() accept the name of a file containing the configuration. However, with some simple Python statements, we can read the json file and convert it into a dictionary. See the example below.
The last option to configure logging is through code.
Configuration using code
All the logging configuration options that are available through configuration files are also possible through the use of source code. Below, I configured the same options as the previous examples but now using code.
Change logging configuration without restarting
To effectively diagnose scenarios such as the one I described in the introduction, you should be able to change the logging level without having to restart the application.
Maybe a particular error only occurs after a certain amount of time. Or you cannot restart the application because it controls hardware such as a heating boiler that takes time to restart.
Default, your application won’t reload the configuration file if you change it. Yet this behavior is possible in Python by creating a logging configuration server.
Logging Configuration Server
To be able to change the logging configuration of a running application, the application needs to implement a logging configuration server. A logging configuration server can be started by calling the listen method on the logging config. Below I configured the logging through code and start a configuration server.
This simple program shows how to start a logging configuration server using logging.config.listen(). The program then starts to log some log messages.
Now, we use a different program to send a new logging configuration to the server listening on port 9000. The new configuration only outputs log messages when they are logged on the critical level. So sending the new configuration will make the logging directly stop as we only log on the debug and info level.
See below for how to send a new logging configuration to the configuration server.
Defining a logger hierarchy
The last feature I want to discuss is creating a logger hierarchy. If you start logging in your code, you will quickly see that it becomes difficult to distinguish individual messages as the number of messages is too large.
It would be helpful if we could enable or disable the logging of certain parts of the application. As we saw previously, online reconfiguration is possible; only did we do this for an entire application. It would be better if we could do this for a specific part of the application.
To make this possible, we can define a hierarchy of loggers.
Each logger that you define or import from third-party libraries is a child of the root logger. You can specify additional levels by using a dot in the name of the logger. For example:
logging.getLogger('a')Here, a is a child of root, b is a child of a, and c is a child of b.
The exciting thing is that when we set the logging level of logger a to INFO, it ripples down the hierarchy. So that the logging level of a.b and a.b.c. is now also INFO. So by composing your loggers using the dot, you create a flexible way to log.
In the example above on row nine, we set the logging of logger a to INFO. But this means that the loggers a.b and a.b.c now also are set to INFO.
Conclusion
The standard logging module in Python is flexible and powerful. In this article, I described how to configure your logger using files and code. I showed you how you could reconfigure your application’s logger without a restart. Finally, I showed how a hierarchy of loggers could help structure your logging.
With my earlier article about the basics of logging in Python, this should give you a running start with logging in Python.
All the source code examples are available on Github
