This tutorial teaches you how to use the Python Logging module to track progress of your Python script and log any errors or exceptions that may arise. Logging information is not enabled by default, so we’ll also show you how to enable Python info logging.

In our earlier Python debugger tutorial, we explained how to keep track of all the variables in your Python script. Python debugger is a handy tool for debugging Python applications in a development environment, but it’s not very useful in production. In production, you can’t stop your application to see what’s causing your application to behave in a certain way. This is where logging comes into play.

With Python logging, you can track your application’s progress and record important information, errors and exceptions your application might face. Even better, you can send these logs to a log file.

Importing the Logging module

To implement logging with Python, you need to import the logging module. The following script imports the logging module and displays its methods, objects, and attributes.

import logging
print(dir(logging))

Output:

['BASIC_FORMAT', 'BufferingFormatter', 'CRITICAL', 'DEBUG', 'ERROR', 'FATAL', 'FileHandler', 'Filter', 'Filterer', 'Formatter', 'Handler', 'INFO', 'LogRecord', 'Logger', 'LoggerAdapter', 'Manager', 'NOTSET', 'NullHandler', 'PercentStyle', 'PlaceHolder', 'RootLogger', 'StrFormatStyle', 'StreamHandler', 'StringTemplateStyle', 'Template', 'WARN', 'WARNING', '_STYLES', '_StderrHandler', '__all__', '__author__', '__builtins__', '__cached__', '__date__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__status__', '__version__', '_acquireLock', '_addHandlerRef', '_checkLevel', '_defaultFormatter', '_defaultLastResort', '_handlerList', '_handlers', '_levelToName', '_lock', '_logRecordFactory', '_loggerClass', '_nameToLevel', '_register_at_fork_reinit_lock', '_releaseLock', '_removeHandlerRef', '_showwarning', '_srcfile', '_startTime', '_str_formatter', '_warnings_showwarning', 'addLevelName', 'atexit', 'basicConfig', 'captureWarnings', 'collections', 'critical', 'currentframe', 'debug', 'disable', 'error', 'exception', 'fatal', 'getLevelName', 'getLogRecordFactory', 'getLogger', 'getLoggerClass', 'handlers', 'info', 'io', 'lastResort', 'log', 'logMultiprocessing', 'logProcesses', 'logThreads', 'makeLogRecord', 'os', 'raiseExceptions', 're', 'root', 'setLogRecordFactory', 'setLoggerClass', 'shutdown', 'sys', 'threading', 'time', 'traceback', 'warn', 'warning', 'warnings', 'weakref']

All the examples in this tutorial require the import logging declaration at the top of your Python script.

Creating and Configuring a Python Logger

To create a log file that will store logging information, you can use the basicConfig() method from the logging module. You need to pass the path to the log file you want to create as a parameter value to the basicConfig() method. Here’s an example.

logging.basicConfig(filename = r"C:\Datasets\mylogger.log")

Your logging file will look like this:

Python Log File

If you open the file, you’ll see that it’s empty. This is because you haven’t actually logged anything in the file yet. If you just enter a file name in basicConfig without a path, it will create the log file in the directory where your script is stored.

The Python logging module supports 5 different levels of log messages. The message levels are DEBUG, INFO, WARNING, ERROR and CRITICAL. Each log message level is assigned a numeric value. The following script prints the numeric value for the 5 message log levels.

print(logging.DEBUG)
print(logging.INFO)
print(logging.WARNING)
print(logging.ERROR)
print(logging.CRITICAL)

Output:

10
20
30
40
50

Now that we know the different logging levels, let’s see what the default log level is for the Python Logging module. To do that, run the following script:

print(logging.getLogger().getEffectiveLevel())

Output:

30

The output shows that the default log level is 30 which means that only the message with log level of 30 or greater will be logged. Since, DEBUG and INFO messages have log levels of 10 and 20, respectively, these messages will not be logged by default. Let’s prove this with an example.

To log messages in your log file, you first need to create a logger object. To do so, you can call the getLogger() method from the logging module as shown in the following script:

my_logger = logging.getLogger()

Next, to log a message, you need to call the method for the message type e.g. debug(), info(), and pass it the message you want to log. For example, the following script is meant to log a DEBUG and an INFO message.

my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")

If you execute the above script and then open your log file, you’ll notice that nothing is printed. This is because the log levels for DEBUG and INFO messages are less than 30, which is the default message log level, so the messages you intended to record in your log file are ignored. To record DEBUG and INFO messages in your Python log file, you need to modify the message log level.

Let’s modify the default message log level and set it to 10. To do so, pass logging.DEBUG as a parameter value to the level attribute of the basicConfig() method as shown below.

logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
                    level = logging.DEBUG,
                    force=True)

print(logging.getLogger().getEffectiveLevel()) #Print result

Output:

10

It’s worth mentioning that the force=True is relatively new to Python. It was first implemented with Python 3.8. Prior to version 3.8, you had to set the default level using setLevel, like this:

logging.basicConfig(filename = r"C:\Datasets\mylogger.log")
logging.getLogger().setLevel(logging.DEBUG)

Using whichever method is compatible with your version of Python, the output proves the default log level is now set to 10. By reducing the default log level, you’re now able to print DEBUG and INFO messages to your log file.

my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")

Now if you open your log file, you’ll see the following two lines in it. The log message by default shows the message type, the logger name, and the message string. The logger name in our case is root since we did not specify any name for the logger.

DEBUG:root:Hello this is debug
INFO:root:Hello this is info

Formatting Log Message Strings

You can format the message string using different logger attributes, which have detailed definitions in the official documentation.

To format your log message, pass a string that specifies the format of your log message to the format attribute of the basicConfig() method.

The following script formats the log message so that it displays the logger name (%name)s, the log time %(asctime)s, and the message itself %(message)s. We didn’t include it on our format_string below, but another useful string modifier is %(levelname)s, which prints the type of error message (DEBUG, INFO, WARNING, ERROR, CRITICAL).

format_string = "%(name)s - %(asctime)s : %(message)s"

logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
                    level = logging.DEBUG, force=True,
                    format = format_string)

Execute the following script to again log the DEBUG and INFO messages.

my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")

If you open your log file now, you should see the following messages with your new format. Notice the last two lines show the logger name, the timestamp for the log entry, and the custom log message you defined. Including a timestamp like this is a smart thing to do when creating log files.

Output:

DEBUG:root:Hello this is debug
INFO:root:Hello this is info
root - 2021-05-05 16:35:01,066 : Hello this is debug
root - 2021-05-05 16:35:01,067 : Hello this is info

The output also shows that log messages are appended at the end of an existing file. The next section explains how you can overwrite your log file whenever a new message is logged.

Overwriting Log Files

To overwrite a log file with new log messages, you need to pass “w” as the parameter value for the filemode parameter of the basicConfig() method of your logger. The log messages in the following script will overwrite all the text in your log file each time it’s executed.

format_string = "%(name)s - %(asctime)s : %(message)s"

logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
                    level = logging.DEBUG, force=True,
                    format = format_string,
                    filemode = 'w')

my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")

If you open your log file now, you’ll see all your previous log messages have been erased and you’re left with your newest log entries.

Output:

root - 2021-05-05 16:41:45,255 : Hello this is debug
root - 2021-05-05 16:41:45,256 : Hello this is info

Get Our Python Developer Kit for Free

I put together a Python Developer Kit with over 100 pre-built Python scripts covering data structures, Pandas, NumPy, Seaborn, machine learning, file processing, web scraping and a whole lot more - and I want you to have it for free. Enter your email address below and I'll send a copy your way.

Yes, I'll take a free Python Developer Kit

A Simple Case of Python Logging

Let’s walk through a very basic use case of Python logging with the help of an example. The following script defines the list_divide() method which returns a list containing element-wise division of the items in two lists. The list_divide() method logs the number of function calls to the method, and records the numerator and denominator lists. If the division fails, an exception is thrown. The exception is also logged using a logger.

import logging
format_string = "%(levelname)s - %(asctime)s : %(message)s"
logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
                    level = logging.DEBUG, force=True,
                    format = format_string,
                    filemode = 'w')
my_logger = logging.getLogger()

i = 0
def list_divide(nums1, nums2):

    global i
    i = i + 1
    my_logger.info("Function called " + str(i) + " times")
    my_logger.info("Numerator list" + str(nums1))
    my_logger.info("Denominator list" + str(nums2))


    try:
        result = []
        for num,den in zip(nums1, nums2):

            result.append(num/den)

        return result
    except Exception as e:
        print("an error occured")
        my_logger.error(e)

If you append the following script to the end of your python file, you won’t see any errors and the output will show the result of element-wise division between the two lists.

list_num = [10, 20, 30, 40, 50]
list_den = [2, 10, 15, 4, 25]

result = list_divide(list_num, list_den)
print(result)

Output:

[5.0, 2.0, 2.0, 10.0, 2.0]

Let’s now add a new list of denominators, list_den, but this time set the second number in the list to 0. Running the script this way will catch an exception.

list_num = [10, 20, 30, 40, 50]
list_den = [2, 0, 15, 4, 25]

result = list_divide(list_num, list_den)
print(result)

Output:

an error occured
None

The exception message is not shown in the output. However, if you open your log file, you’ll see the following text.

INFO - 2021-05-22 08:28:17,189 : Function called 1 times
INFO - 2021-05-22 08:28:17,189 : Numerator list[10, 20, 30, 40, 50]
INFO - 2021-05-22 08:28:17,189 : Denominator list[2, 10, 15, 4, 25]
INFO - 2021-05-22 08:28:17,201 : Function called 2 times
INFO - 2021-05-22 08:28:17,201 : Numerator list[10, 20, 30, 40, 50]
INFO - 2021-05-22 08:28:17,201 : Denominator list[2, 0, 15, 4, 25]
ERROR - 2021-05-22 08:28:17,207 : division by zero

You can see that on the second call to the list_divide() function, a division by zero error occurred. By thoroughly reviewing the log file, you’ll recognize the denominator list contains an item with a value of 0. With good info and error log entries like this, you’re now able to debug your Python code in production using the Python logging module.


Get Our Python Developer Kit for Free

I put together a Python Developer Kit with over 100 pre-built Python scripts covering data structures, Pandas, NumPy, Seaborn, machine learning, file processing, web scraping and a whole lot more - and I want you to have it for free. Enter your email address below and I'll send a copy your way.

Yes, I'll take a free Python Developer Kit