Machine Learning Interview: Computer Vision
2021-07-02
How to use black, flake8, isort, and pre-commit framework to format Python codes
2021-08-03
Show all

Best practices for Python exceptions

9 mins read

Python is a popular and versatile programming language used in a wide range of applications. One of the most important aspects of writing robust and reliable Python code is handling exceptions effectively. Exceptions are unexpected events that can occur during the execution of a program, such as errors, warnings, or other unexpected conditions. In order to handle these exceptions, it is important to follow best practices and established conventions to ensure that your code is readable, maintainable, and free from bugs. In this blog post, we will explore some of the best practices for handling Python exceptions, including how to raise and catch exceptions, how to handle errors gracefully, and how to create custom exceptions for your specific use case. By following these best practices, you can write Python code that is more reliable, easier to debug, and better suited to your particular needs.

How do I manually throw/raise an exception in Python?

Use the most specific Exception constructor that semantically fits your issue.

Be specific in your message, e.g.:

raise ValueError('A very specific bad thing happened.')

Don’t raise generic exceptions

Avoid raising a generic Exception. To catch it, you’ll have to catch all other more specific exceptions that subclass it.

Problem 1: Hiding bugs

raise Exception('I know Python!') # Don't! If you catch, likely to hide bugs.

For example:

def demo_bad_catch():
    try:
        raise ValueError('Represents a hidden bug, do not catch this')
        raise Exception('This is the exception you expect to handle')
    except Exception as error:
        print('Caught this error: ' + repr(error))

>>> demo_bad_catch()
Caught this error: ValueError('Represents a hidden bug, do not catch this',)

Problem 2: Won’t catch

and more specific catches won’t catch the general exception:

def demo_no_catch():
    try:
        raise Exception('general exceptions not caught by specific handling')
    except ValueError as e:
        print('we will not catch exception: Exception')


>>> demo_no_catch()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in demo_no_catch
Exception: general exceptions not caught by specific handling

Best Practices: raise statement

Instead, use the most specific Exception constructor that semantically fits your issue.

raise ValueError('A very specific bad thing happened')

which also handily allows an arbitrary number of arguments to be passed to the constructor:

raise ValueError('A very specific bad thing happened', 'foo', 'bar', 'baz') 

These arguments are accessed by the args attribute on the Exception object. For example:

try:
    some_code_that_may_raise_our_value_error()
except ValueError as err:
    print(err.args)

prints

('message', 'foo', 'bar', 'baz')    

In Python 2.5, an actual message attribute was added to BaseException in favor of encouraging users to subclass Exceptions and stop using args, but the introduction of message and the original deprecation of args has been retracted.

Best Practices: except clause

When inside an except clause, you might want to, for example, log that a specific type of error happened, and then re-raise. The best way to do this while preserving the stack trace is to use a bare raise statement. For example:

logger = logging.getLogger(__name__)

try:
    do_something_in_app_that_breaks_easily()
except AppError as error:
    logger.error(error)
    raise                 # just this!
    # raise AppError      # Don't do this, you'll lose the stack trace!

Don’t modify your errors… but if you insist.

You can preserve the stack trace (and error value) with sys.exc_info(), but this is way more error-prone and has compatibility problems between Python 2 and 3, prefer to use a bare raise to re-raise.

To explain – the sys.exc_info() returns the type, value, and traceback.

type, value, traceback = sys.exc_info()

This is the syntax in Python 2 – note this is not compatible with Python 3:

    raise AppError, error, sys.exc_info()[2] # avoid this.
    # Equivalently, as error *is* the second object:
    raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]

If you want to, you can modify what happens with your new raise – e.g. setting new args for the instance:

def error():
    raise ValueError('oops!')

def catch_error_modify_message():
    try:
        error()
    except ValueError:
        error_type, error_instance, traceback = sys.exc_info()
        error_instance.args = (error_instance.args[0] + ' <modification>',)
        raise error_type, error_instance, traceback

And we have preserved the whole traceback while modifying the args. Note that this is not a best practice and it is invalid syntax in Python 3 (making keeping compatibility much harder to workaround).

>>> catch_error_modify_message()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in catch_error_modify_message
  File "<stdin>", line 2, in error
ValueError: oops! <modification>

In Python 3:

    raise error.with_traceback(sys.exc_info()[2])

Again: avoid manually manipulating tracebacks. It’s less efficient and more error-prone. And if you’re using threading and sys.exc_info you may even get the wrong traceback (especially if you’re using exception handling for control flow – which I’d personally tend to avoid.)

Python 3, Exception chaining

In Python 3, you can chain Exceptions, which preserve tracebacks:

    raise RuntimeError('specific message') from error

Be aware:

  • this does allow changing the error type raised, and
  • this is not compatible with Python 2.

Deprecated Methods:

These can easily hide and even get into production code. You want to raise an exception, and doing them will raise an exception, but not the one intended!

Valid in Python 2, but not in Python 3 is the following:

raise ValueError, 'message' # Don't do this, it's deprecated!

Only valid in much older versions of Python (2.4 and lower), you may still see people raising strings:

raise 'message' # really really wrong. don't do this.

In all modern versions, this will actually raise a TypeError, because you’re not raising a BaseException type. If you’re not checking for the right exception and don’t have a reviewer that’s aware of the issue, it could get into production.

Example Usage

I raise Exceptions to warn consumers of my API if they’re using it incorrectly:

def api_func(foo):
    '''foo should be either 'baz' or 'bar'. returns something very useful.'''
    if foo not in _ALLOWED_ARGS:
        raise ValueError('{foo} wrong, use "baz" or "bar"'.format(foo=repr(foo)))

Create your own error types when apropos

I want to make an error on purpose so that it would go into the except”

You can create your own error types, if you want to indicate something specific is wrong with your application, just subclass the appropriate point in the exception hierarchy:

class MyAppLookupError(LookupError):
    '''raise this when there's a lookup error for my app'''

and usage:

if important_key not in resource_dict and not ok_to_be_missing:
    raise MyAppLookupError('resource is missing, and that is not ok.')

Using try/except vs. if/else in python

Is there a rationale to decide which one of try or if constructs to use when testing variables to have a value?

For example, there is a function that returns either a list or doesn’t return a value. I want to check the result before processing it. Which of the following would be more preferable and why?

result = function();
if (result):
    for r in result:
        #process items

or

result = function();
try:
    for r in result:
        #process items
except TypeError:
    pass;

You often hear that Python encourages EAFP style (“it’s easier to ask for forgiveness than permission”) over LBYL style (“look before you leap”). To me, it’s a matter of efficiency and readability.

In your example (say that instead of returning a list or an empty string, the function was to return a list or None), if you expect that 99 % of the time result will actually contain something iterable, I’d use the try/except approach. It will be faster if exceptions really are exceptional. If result is None more than 50 % of the time, then using if is probably better.

So, whereas an if statement always costs you, it’s nearly free to set up a try/except block. But when an Exception actually occurs, the cost is much higher.

Moral:

  • It’s perfectly OK (and “pythonic”) to use try/except for flow control,
  • but it makes sense most when Exceptions are actually exceptional.

Using Specialized exception class

Terminating with sys.exit might be considered bad form in python: exceptions are the proper way to generate/handle errors. To differentiate application-level exceptions from other python exceptions, we create a specialized class, which inherits from python’s Exception class. Throw it when there is an application-level error, and catch it in the main code. This will also be more modular if we ever want to convert this code into a python module (a python module should never call sys.exit).

#!/usr/bin/env python
import sys, random

class MyException(Exception):
    """ My Class for user-facing (non-programming) errors """
    pass

def generate_runtime_error():
    # Raise I/O error
    try:
        a = open("/non/existing/file","r")
    except IOError as e:
        err = "Input/Output error: %s" % ( str(e) )
        raise MyException(err)

def generate_value_error():
    filename = "/etc/passwd"
    try:
        a = open(filename,"r")
        # Get the username of the first user (usually 'root')
        b = a.readline().split(":")[0]
        # Convert to a number - this will fail with ValueError exception
        c = int(b)
    except IOError as e:
        err = "Input/Output error: %s" % ( str(e) )
        raise MyException(err)
    except ValueError as e:
        err = "input validation error in '%s': " \
                 "expecting numeric value, but found '%s'" % \
                 ( filename, b )
        raise MyException(err)

def generate_attribute_error():
    a = open("/etc/passwd","r")
    # Programming error: 'a' doesn't have function 'readdline'
    # (type with two 'd')
    # will raise AttributeError
    b = a.readdline().split(":")[0]
    c = int(b)


def generate_type_error():
    # Invalid Python code, will raise TypeError exception
    a = "hello"
    b = "%s %s %s " % ( a )

try:
    r = random.randint(1,4)
    if r==1:
        generate_runtime_error()
    elif r==2:
        generate_value_error()
    elif r==3:
        generate_attribute_error()
    else:
        generate_type_error()
except MyException as e:
    # Centralized place for termination-cleanup
    sys.exit("program failed: " + str(e))

Other Points:

If a try statement has several except clauses, the exception propagation mechanism tests the except clauses in order: the first except clause whose expression matches the exception object is used as the handler. Thus, you must always list handlers for specific cases before you list handlers for more general cases. If you list a general case first, the more specific except clauses that follow will never enter the picture.

The last except clause may lack an expression. This clause handles any exception that reaches it during propagation. Such unconditional handling is a rare need, but it does occur, generally in wrapper functions that must perform some extra task before reraising an exception.

if a try statement is nested in the try clause of another try statement, a handler established by the inner try is reached first during propagation and therefore is the one that handles the exception, if it matches the expression. For example:

try:
    try: 1/0
    except: print "caught an exception"
except ZeroDivisionError:
    print "caught divide-by-0 attempt"
# prints: caught an exception

In this case, it does not matter that the handler established by clause except ZeroDivisionError: in the outer try clause is more specific and appropriate than the catch-all except: in the inner try clause. The outer try does not even enter into the picture because the exception doesn’t propagate out of the inner try.

The optional else clause of try/except executes only if the try clause terminates normally. In other words, it does not execute if an exception propagates from the try clause or if the try clause exits with a break, continue, or return statement. The handlers established by try/except cover only the try clause, not the else clause.

Catching exceptions as close to the origin as possible

Catching an error in the ‘main’ section might miss some information (e.g. “ValueError” could originate from many different places in the code from many different inputs). It might be considered better to catch the error as close as possible to the source (but – only user/runtime errors, not programming errors)

Useful Links:

https://docs.python.org/3/tutorial/errors.html

https://www.pythonforthelab.com/blog/learning-not-to-handle-exceptions/

https://crashcourse.housegordon.org/python-exceptions-handling-tips.html

https://realpython.com/python-exceptions/

https://www.datacamp.com/community/tutorials/exception-handling-python

http://www.ianbicking.org/blog/2007/09/re-raising-exceptions.html

https://stackoverflow.com/questions/2052390/manually-raising-throwing-an-exception-in-python

http://etutorials.org/Programming/Python+tutorial/Part+II+Core+Python+Language+and+Built-ins/Chapter+6.+Exceptions/

https://github.com/PFTL/website/blob/master/content/blog/12_handling_exceptions.rst

1 Comment

  1. […] Best practices to handle Python Exceptions – Amir Masoud (sefidian.com) […]

Leave a Reply to Handle Exceptions and Errors with Python traceback and sys Modules – vegibit Cancel reply

Your email address will not be published. Required fields are marked *