Python is a popular and versatile programming language used in a wide range of applications. One of the most important aspects of writing robust and reliable Python code is handling exceptions effectively. Exceptions are unexpected events that can occur during the execution of a program, such as errors, warnings, or other unexpected conditions. In order to handle these exceptions, it is important to follow best practices and established conventions to ensure that your code is readable, maintainable, and free from bugs. In this blog post, we will explore some of the best practices for handling Python exceptions, including how to raise and catch exceptions, how to handle errors gracefully, and how to create custom exceptions for your specific use case. By following these best practices, you can write Python code that is more reliable, easier to debug, and better suited to your particular needs.
Use the most specific Exception constructor that semantically fits your issue.
Be specific in your message, e.g.:
raise ValueError('A very specific bad thing happened.')
Avoid raising a generic Exception. To catch it, you’ll have to catch all other more specific exceptions that subclass it.
raise Exception('I know Python!') # Don't! If you catch, likely to hide bugs.
For example:
def demo_bad_catch():
try:
raise ValueError('Represents a hidden bug, do not catch this')
raise Exception('This is the exception you expect to handle')
except Exception as error:
print('Caught this error: ' + repr(error))
>>> demo_bad_catch()
Caught this error: ValueError('Represents a hidden bug, do not catch this',)
and more specific catches won’t catch the general exception:
def demo_no_catch():
try:
raise Exception('general exceptions not caught by specific handling')
except ValueError as e:
print('we will not catch exception: Exception')
>>> demo_no_catch()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in demo_no_catch
Exception: general exceptions not caught by specific handling
raise
statementInstead, use the most specific Exception constructor that semantically fits your issue.
raise ValueError('A very specific bad thing happened')
which also handily allows an arbitrary number of arguments to be passed to the constructor:
raise ValueError('A very specific bad thing happened', 'foo', 'bar', 'baz')
These arguments are accessed by the args
attribute on the Exception object. For example:
try:
some_code_that_may_raise_our_value_error()
except ValueError as err:
print(err.args)
prints
('message', 'foo', 'bar', 'baz')
In Python 2.5, an actual message
attribute was added to BaseException in favor of encouraging users to subclass Exceptions and stop using args
, but the introduction of message
and the original deprecation of args has been retracted.
except
clauseWhen inside an except clause, you might want to, for example, log that a specific type of error happened, and then re-raise. The best way to do this while preserving the stack trace is to use a bare raise statement. For example:
logger = logging.getLogger(__name__)
try:
do_something_in_app_that_breaks_easily()
except AppError as error:
logger.error(error)
raise # just this!
# raise AppError # Don't do this, you'll lose the stack trace!
You can preserve the stack trace (and error value) with sys.exc_info(), but this is way more error-prone and has compatibility problems between Python 2 and 3, prefer to use a bare raise
to re-raise.
To explain – the sys.exc_info()
returns the type, value, and traceback.
type, value, traceback = sys.exc_info()
This is the syntax in Python 2 – note this is not compatible with Python 3:
raise AppError, error, sys.exc_info()[2] # avoid this.
# Equivalently, as error *is* the second object:
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
If you want to, you can modify what happens with your new raise – e.g. setting new args for the instance:
def error():
raise ValueError('oops!')
def catch_error_modify_message():
try:
error()
except ValueError:
error_type, error_instance, traceback = sys.exc_info()
error_instance.args = (error_instance.args[0] + ' <modification>',)
raise error_type, error_instance, traceback
And we have preserved the whole traceback while modifying the args. Note that this is not a best practice and it is invalid syntax in Python 3 (making keeping compatibility much harder to workaround).
>>> catch_error_modify_message()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in catch_error_modify_message
File "<stdin>", line 2, in error
ValueError: oops! <modification>
In Python 3:
raise error.with_traceback(sys.exc_info()[2])
Again: avoid manually manipulating tracebacks. It’s less efficient and more error-prone. And if you’re using threading and sys.exc_info
you may even get the wrong traceback (especially if you’re using exception handling for control flow – which I’d personally tend to avoid.)
In Python 3, you can chain Exceptions, which preserve tracebacks:
raise RuntimeError('specific message') from error
Be aware:
These can easily hide and even get into production code. You want to raise an exception, and doing them will raise an exception, but not the one intended!
Valid in Python 2, but not in Python 3 is the following:
raise ValueError, 'message' # Don't do this, it's deprecated!
Only valid in much older versions of Python (2.4 and lower), you may still see people raising strings:
raise 'message' # really really wrong. don't do this.
In all modern versions, this will actually raise a TypeError, because you’re not raising a BaseException type. If you’re not checking for the right exception and don’t have a reviewer that’s aware of the issue, it could get into production.
I raise Exceptions to warn consumers of my API if they’re using it incorrectly:
def api_func(foo):
'''foo should be either 'baz' or 'bar'. returns something very useful.'''
if foo not in _ALLOWED_ARGS:
raise ValueError('{foo} wrong, use "baz" or "bar"'.format(foo=repr(foo)))
“I want to make an error on purpose so that it would go into the except”
You can create your own error types, if you want to indicate something specific is wrong with your application, just subclass the appropriate point in the exception hierarchy:
class MyAppLookupError(LookupError):
'''raise this when there's a lookup error for my app'''
and usage:
if important_key not in resource_dict and not ok_to_be_missing:
raise MyAppLookupError('resource is missing, and that is not ok.')
Is there a rationale to decide which one of try
or if
constructs to use when testing variables to have a value?
For example, there is a function that returns either a list or doesn’t return a value. I want to check the result before processing it. Which of the following would be more preferable and why?
result = function();
if (result):
for r in result:
#process items
or
result = function();
try:
for r in result:
#process items
except TypeError:
pass;
You often hear that Python encourages EAFP style (“it’s easier to ask for forgiveness than permission”) over LBYL style (“look before you leap”). To me, it’s a matter of efficiency and readability.
In your example (say that instead of returning a list or an empty string, the function was to return a list or None
), if you expect that 99 % of the time result
will actually contain something iterable, I’d use the try/except
approach. It will be faster if exceptions really are exceptional. If result
is None
more than 50 % of the time, then using if
is probably better.
So, whereas an if
statement always costs you, it’s nearly free to set up a try/except
block. But when an Exception
actually occurs, the cost is much higher.
Moral:
try/except
for flow control,Exception
s are actually exceptional.Terminating with sys.exit
might be considered bad form in python: exceptions are the proper way to generate/handle errors. To differentiate application-level exceptions from other python exceptions, we create a specialized class, which inherits from python’s Exception
class. Throw it when there is an application-level error, and catch it in the main code. This will also be more modular if we ever want to convert this code into a python module (a python module should never call sys.exit
).
#!/usr/bin/env python
import sys, random
class MyException(Exception):
""" My Class for user-facing (non-programming) errors """
pass
def generate_runtime_error():
# Raise I/O error
try:
a = open("/non/existing/file","r")
except IOError as e:
err = "Input/Output error: %s" % ( str(e) )
raise MyException(err)
def generate_value_error():
filename = "/etc/passwd"
try:
a = open(filename,"r")
# Get the username of the first user (usually 'root')
b = a.readline().split(":")[0]
# Convert to a number - this will fail with ValueError exception
c = int(b)
except IOError as e:
err = "Input/Output error: %s" % ( str(e) )
raise MyException(err)
except ValueError as e:
err = "input validation error in '%s': " \
"expecting numeric value, but found '%s'" % \
( filename, b )
raise MyException(err)
def generate_attribute_error():
a = open("/etc/passwd","r")
# Programming error: 'a' doesn't have function 'readdline'
# (type with two 'd')
# will raise AttributeError
b = a.readdline().split(":")[0]
c = int(b)
def generate_type_error():
# Invalid Python code, will raise TypeError exception
a = "hello"
b = "%s %s %s " % ( a )
try:
r = random.randint(1,4)
if r==1:
generate_runtime_error()
elif r==2:
generate_value_error()
elif r==3:
generate_attribute_error()
else:
generate_type_error()
except MyException as e:
# Centralized place for termination-cleanup
sys.exit("program failed: " + str(e))
If a try statement has several except clauses, the exception propagation mechanism tests the except clauses in order: the first except clause whose expression matches the exception object is used as the handler. Thus, you must always list handlers for specific cases before you list handlers for more general cases. If you list a general case first, the more specific except clauses that follow will never enter the picture.
The last except clause may lack an expression. This clause handles any exception that reaches it during propagation. Such unconditional handling is a rare need, but it does occur, generally in wrapper functions that must perform some extra task before reraising an exception.
if a try statement is nested in the try clause of another try statement, a handler established by the inner try is reached first during propagation and therefore is the one that handles the exception, if it matches the expression. For example:
try:
try: 1/0
except: print "caught an exception"
except ZeroDivisionError:
print "caught divide-by-0 attempt"
# prints: caught an exception
In this case, it does not matter that the handler established by clause except ZeroDivisionError: in the outer try clause is more specific and appropriate than the catch-all except: in the inner try clause. The outer try does not even enter into the picture because the exception doesn’t propagate out of the inner try.
The optional else clause of try/except executes only if the try clause terminates normally. In other words, it does not execute if an exception propagates from the try clause or if the try clause exits with a break, continue, or return statement. The handlers established by try/except cover only the try clause, not the else clause.
Catching an error in the ‘main’ section might miss some information (e.g. “ValueError” could originate from many different places in the code from many different inputs). It might be considered better to catch the error as close as possible to the source (but – only user/runtime errors, not programming errors)
https://docs.python.org/3/tutorial/errors.html
https://www.pythonforthelab.com/blog/learning-not-to-handle-exceptions/
https://crashcourse.housegordon.org/python-exceptions-handling-tips.html
https://realpython.com/python-exceptions/
https://www.datacamp.com/community/tutorials/exception-handling-python
http://www.ianbicking.org/blog/2007/09/re-raising-exceptions.html
https://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python
https://github.com/PFTL/website/blob/master/content/blog/12_handling_exceptions.rst
1 Comment
[…] Best practices to handle Python Exceptions – Amir Masoud (sefidian.com) […]