This tutorial teaches you how to use the Python Logging module to track progress of your Python script and log any errors or exceptions that may arise. Logging information is not enabled by default, so we’ll also show you how to enable Python info logging.
In our earlier Python debugger tutorial, we explained how to keep track of all the variables in your Python script. Python debugger is a handy tool for debugging Python applications in a development environment, but it’s not very useful in production. In production, you can’t stop your application to see what’s causing your application to behave in a certain way. This is where logging comes into play.
With Python logging, you can track your application’s progress and record important information, errors and exceptions your application might face. Even better, you can send these logs to a log file.
Importing the Logging module
To implement logging with Python, you need to import the logging module. The following script imports the logging module and displays its methods, objects, and attributes.
import logging
print(dir(logging))Output:
['BASIC_FORMAT', 'BufferingFormatter', 'CRITICAL', 'DEBUG', 'ERROR', 'FATAL', 'FileHandler', 'Filter', 'Filterer', 'Formatter', 'Handler', 'INFO', 'LogRecord', 'Logger', 'LoggerAdapter', 'Manager', 'NOTSET', 'NullHandler', 'PercentStyle', 'PlaceHolder', 'RootLogger', 'StrFormatStyle', 'StreamHandler', 'StringTemplateStyle', 'Template', 'WARN', 'WARNING', '_STYLES', '_StderrHandler', '__all__', '__author__', '__builtins__', '__cached__', '__date__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__status__', '__version__', '_acquireLock', '_addHandlerRef', '_checkLevel', '_defaultFormatter', '_defaultLastResort', '_handlerList', '_handlers', '_levelToName', '_lock', '_logRecordFactory', '_loggerClass', '_nameToLevel', '_register_at_fork_reinit_lock', '_releaseLock', '_removeHandlerRef', '_showwarning', '_srcfile', '_startTime', '_str_formatter', '_warnings_showwarning', 'addLevelName', 'atexit', 'basicConfig', 'captureWarnings', 'collections', 'critical', 'currentframe', 'debug', 'disable', 'error', 'exception', 'fatal', 'getLevelName', 'getLogRecordFactory', 'getLogger', 'getLoggerClass', 'handlers', 'info', 'io', 'lastResort', 'log', 'logMultiprocessing', 'logProcesses', 'logThreads', 'makeLogRecord', 'os', 'raiseExceptions', 're', 'root', 'setLogRecordFactory', 'setLoggerClass', 'shutdown', 'sys', 'threading', 'time', 'traceback', 'warn', 'warning', 'warnings', 'weakref']All the examples in this tutorial require the import logging declaration at the top of your Python script.
Creating and Configuring a Python Logger
To create a log file that will store logging information, you can use the basicConfig() method from the logging module. You need to pass the path to the log file you want to create as a parameter value to the basicConfig() method. Here’s an example.
logging.basicConfig(filename = r"C:\Datasets\mylogger.log")Your logging file will look like this:
If you open the file, you’ll see that it’s empty. This is because you haven’t actually logged anything in the file yet. If you just enter a file name in basicConfig without a path, it will create the log file in the directory where your script is stored.
The Python logging module supports 5 different levels of log messages. The message levels are DEBUG, INFO, WARNING, ERROR and CRITICAL. Each log message level is assigned a numeric value. The following script prints the numeric value for the 5 message log levels.
print(logging.DEBUG)
print(logging.INFO)
print(logging.WARNING)
print(logging.ERROR)
print(logging.CRITICAL)Output:
10
20
30
40
50Now that we know the different logging levels, let’s see what the default log level is for the Python Logging module. To do that, run the following script:
print(logging.getLogger().getEffectiveLevel())Output:
30The output shows that the default log level is 30 which means that only the message with log level of 30 or greater will be logged. Since, DEBUG and INFO messages have log levels of 10 and 20, respectively, these messages will not be logged by default. Let’s prove this with an example.
To log messages in your log file, you first need to create a logger object. To do so, you can call the getLogger() method from the logging module as shown in the following script:
my_logger = logging.getLogger()Next, to log a message, you need to call the method for the message type e.g. debug(), info(), and pass it the message you want to log. For example, the following script is meant to log a DEBUG and an INFO message.
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")If you execute the above script and then open your log file, you’ll notice that nothing is printed. This is because the log levels for DEBUG and INFO messages are less than 30, which is the default message log level, so the messages you intended to record in your log file are ignored. To record DEBUG and INFO messages in your Python log file, you need to modify the message log level.
Let’s modify the default message log level and set it to 10. To do so, pass logging.DEBUG as a parameter value to the level attribute of the basicConfig() method as shown below.
logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
level = logging.DEBUG,
force=True)
print(logging.getLogger().getEffectiveLevel()) #Print resultOutput:
10It’s worth mentioning that the force=True is relatively new to Python. It was first implemented with Python 3.8. Prior to version 3.8, you had to set the default level using setLevel, like this:
logging.basicConfig(filename = r"C:\Datasets\mylogger.log")
logging.getLogger().setLevel(logging.DEBUG)Using whichever method is compatible with your version of Python, the output proves the default log level is now set to 10. By reducing the default log level, you’re now able to print DEBUG and INFO messages to your log file.
my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")Now if you open your log file, you’ll see the following two lines in it. The log message by default shows the message type, the logger name, and the message string. The logger name in our case is root since we did not specify any name for the logger.
DEBUG:root:Hello this is debug
INFO:root:Hello this is infoFormatting Log Message Strings
You can format the message string using different logger attributes, which have detailed definitions in the official documentation.
To format your log message, pass a string that specifies the format of your log message to the format attribute of the basicConfig() method.
The following script formats the log message so that it displays the logger name
format_string = "%(name)s - %(asctime)s : %(message)s"
logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
level = logging.DEBUG, force=True,
format = format_string)Execute the following script to again log the DEBUG and INFO messages.
my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")If you open your log file now, you should see the following messages with your new format. Notice the last two lines show the logger name, the timestamp for the log entry, and the custom log message you defined. Including a timestamp like this is a smart thing to do when creating log files.
Output:
DEBUG:root:Hello this is debug
INFO:root:Hello this is info
root - 2021-05-05 16:35:01,066 : Hello this is debug
root - 2021-05-05 16:35:01,067 : Hello this is infoThe output also shows that log messages are appended at the end of an existing file. The next section explains how you can overwrite your log file whenever a new message is logged.
Overwriting Log Files
To overwrite a log file with new log messages, you need to pass “w” as the parameter value for the filemode parameter of the basicConfig() method of your logger. The log messages in the following script will overwrite all the text in your log file each time it’s executed.
format_string = "%(name)s - %(asctime)s : %(message)s"
logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
level = logging.DEBUG, force=True,
format = format_string,
filemode = 'w')
my_logger = logging.getLogger()
my_logger.debug("Hello this is debug")
my_logger.info("Hello this is info")If you open your log file now, you’ll see all your previous log messages have been erased and you’re left with your newest log entries.
Output:
root - 2021-05-05 16:41:45,255 : Hello this is debug
root - 2021-05-05 16:41:45,256 : Hello this is infoCode More, Distract Less: Support Our Ad-Free Site
You might have noticed we removed ads from our site - we hope this enhances your learning experience. To help sustain this, please take a look at our Python Developer Kit and our comprehensive cheat sheets. Each purchase directly supports this site, ensuring we can continue to offer you quality, distraction-free tutorials.
A Simple Case of Python Logging
Let’s walk through a very basic use case of Python logging with the help of an example. The following script defines the list_divide() method which returns a list containing element-wise division of the items in two lists. The list_divide() method logs the number of function calls to the method, and records the numerator and denominator lists. If the division fails, an exception is thrown. The exception is also logged using a logger.
import logging
format_string = "%(levelname)s - %(asctime)s : %(message)s"
logging.basicConfig(filename = r"C:\Datasets\mylogger.log",
level = logging.DEBUG, force=True,
format = format_string,
filemode = 'w')
my_logger = logging.getLogger()
i = 0
def list_divide(nums1, nums2):
global i
i = i + 1
my_logger.info("Function called " + str(i) + " times")
my_logger.info("Numerator list" + str(nums1))
my_logger.info("Denominator list" + str(nums2))
try:
result = []
for num,den in zip(nums1, nums2):
result.append(num/den)
return result
except Exception as e:
print("an error occured")
my_logger.error(e)If you append the following script to the end of your python file, you won’t see any errors and the output will show the result of element-wise division between the two lists.
list_num = [10, 20, 30, 40, 50]
list_den = [2, 10, 15, 4, 25]
result = list_divide(list_num, list_den)
print(result)Output:
[5.0, 2.0, 2.0, 10.0, 2.0]Let’s now add a new list of denominators,
list_num = [10, 20, 30, 40, 50]
list_den = [2, 0, 15, 4, 25]
result = list_divide(list_num, list_den)
print(result)Output:
an error occured
NoneThe exception message is not shown in the output. However, if you open your log file, you’ll see the following text.
INFO - 2021-05-22 08:28:17,189 : Function called 1 times
INFO - 2021-05-22 08:28:17,189 : Numerator list[10, 20, 30, 40, 50]
INFO - 2021-05-22 08:28:17,189 : Denominator list[2, 10, 15, 4, 25]
INFO - 2021-05-22 08:28:17,201 : Function called 2 times
INFO - 2021-05-22 08:28:17,201 : Numerator list[10, 20, 30, 40, 50]
INFO - 2021-05-22 08:28:17,201 : Denominator list[2, 0, 15, 4, 25]
ERROR - 2021-05-22 08:28:17,207 : division by zeroYou can see that on the second call to the list_divide() function, a division by zero error occurred. By thoroughly reviewing the log file, you’ll recognize the denominator list contains an item with a value of 0. With good info and error log entries like this, you’re now able to debug your Python code in production using the Python logging module.
Code More, Distract Less: Support Our Ad-Free Site
You might have noticed we removed ads from our site - we hope this enhances your learning experience. To help sustain this, please take a look at our Python Developer Kit and our comprehensive cheat sheets. Each purchase directly supports this site, ensuring we can continue to offer you quality, distraction-free tutorials.
