Introducing the New Features in Python 3.11

We’re Earthly. We make building software simpler and therefore faster using containerization. Excited about Python 3.11’s new features? Earthly can make your Python builds even more efficient. Check it out.

Python 3.11 is the latest version of the Python programming language. It includes a variety of new features and performance enhancement.

This article will introduce you to some of the new features of Python 3.11. These features include:

• Better error handling
• Improved type annotation
• A new built-in library for working with TOML files
• Improved speed performance

While this tutorial will provide an overview of these features, I will also provide additional resources for further learning when necessary.

To follow along with this tutorial, you should have a basic understanding of Python. You will also need to have Python 3.11 installed. You can download it for your specific operating system from the official website.

All code examples used in this tutorial can be found in this GitHub repository.

Error and Exception Handling Features

One of the major improvements added in the new version of Python centers around exception handling and helpful traceback when exceptions arise.

Traceback Annotation

To enhance your debugging experience, the new Python version includes a traceback annotation feature.

When an error occurs, the Python interpreter will now indicate the specific part of the code that produced the error, rather than just the line where the error occurred. This can be particularly helpful, as it can sometimes be difficult to locate an error based on only the line number.

Take the following code snippet for instance. The code snippet creates an Article class with a title and an optional author attribute. The author attribute defaults to None:

class Article:
   def __init__(self, title, author=None) -> None:
       self.title = title
       self.author = author

article_1 = Article(title="Introducing Python 3.11", author="Ahmad")
article_2 = Article(title="Python runtime speed enhanced")
article_3 = Article(title="Enhance Python Error", author="Mustapha")

print(article_1.title.upper(), article_2.author.upper(), \
article_3.author.upper())

If you execute the code above with the Python 3.10 interpreter (or lower), it will generate the following output:

Traceback (most recent call last):
  File "/home/dracode/python-11/main.py", line 26, in <module>
    print(article_1.title.upper(), article_2.author.upper(), \
    article_3.author.upper())
AttributeError: 'NoneType' object has no attribute 'upper'

This error message is ambiguous as the interpreter does not indicate which of the instances you called the .upper() method on has a NoneType. The error could be from any of the three instances; article_1, article_2 and article_3.

The new Python 3.11 interpreter will produce the following output:

Traceback (most recent call last):
  File "/home/dracode/python-11/main.py", line 26, in <module>
    print(article_1.title.upper(), article_2.author.upper(), \
    article_3.author.upper())
                                           ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'upper'

The interpreter indicated the exact part of the code that produced the error.

The older versions of interpreters determine the line that produces the error from the bytecode that Python generates after compiling the code. Python maps bytecode to the line number after code compilation. These bytecodes are saved in the .pyc files. This new feature adds more information to the bytecode that can be used to point to the exact location of the error as proposed in the PEP 657.

This new information in the bytecode will have an impact on the size of pyc files on disk and the size of code objects in memory. If you are concerned about this memory overhead, the traceback annotation feature is an optional feature that you can opt out of by setting the PYTHONNODEBUGRANGES environmental variable to False or executing the following command line option:

$ python -Xno_debug_ranges

The ExceptionGroup class and the except* Statement

The new version of Python introduces an Exception group and except* syntax for working with Exception groups.

Exception groups let you group related exceptions together.

Exceptions are a break from the normal flow of a program that indicates that an error has occurred. These exceptions terminate a program unless they are handled appropriately.

I will give a cursory overview of these exceptions so that you can understand how they work differently than the new ExceptionGroup and the new except * statement.

The exception classes are a subclass of Python’s BaseException class. Python has a long list of built-in exception classes. Python raises them when it doesn’t understand a syntax or whenever it encounters an invalid operation or invalid inputs in your code. You can also raise them explicitly with the raise keyword, as shown below:

raise SyntaxError("Just raising a syntax error")

Output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SyntaxError: Just raising a syntax error

These exceptions take an optional message argument that is meant to give more information about the error.

They terminate a program when they are encountered, but you can handle them when you anticipate them. Python has a try and except statement that allows you to handle them so that they do not terminate the program.

try:
   print(34/0)
except ZeroDivisionError:
   print("You can't divide a value by 0, how about try dividing by 2?")
   print(34/2)

Output:

You can't divide a value by 0, how about trying dividing by 2?
17

The try statement can have more than one except block that handles different exceptions, however, Python only executes the first except block that matches the exception and ignores the other except blocks.

try:
   prin(34/2)
except NameError:
   print("There is an undefined name up there, take a look at it again!")
except SyntaxError:
   print("hey! There could be a syntax error there!")

Output:

There is an undefined name up there, take a look at it again!

In the example above, Python only executes the except block that handles the NameError that matches the exception raised. It doesn’t bother checking the other except blocks.

You can specify multiple exception classes in the except statement. Python executes the block if any of the exception classes match the error:

try:
   prin(34/0)
except (ZeroDivisionError, NameError) as exc:
   print(exc)

Output:

name 'prin' is not defined

While handling an exception in a try and except block, another unhandled exception might propagate. Python’s exception chaining feature (which was introduced in PEP 3134) allows Python to show the unhandled exception that propagates while handling another exception.

The code below raises a NameError exception due to the undefined prin function (which you handled), then raises a new ZeroDivisionError due to dividing 3 by 0 in the except block:

try:
    prin(3/0)
except NameError:
    print(3/0)

Instead of silencing the exception you handled, Python indicates:

• The exception that it raises while you were handling another exception
• The exception that you handled.

This is indicated by the During handling of the above exception, another exception occurred statement in the traceback below :

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: name 'prin' is not defined. Did you mean: 'print'?

During the handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
ZeroDivisionError: division by zero

The try and except statement allows you to handle any type of exception class, however, it is limited because:

• Only a single exception is handled at a time
• It only executes the first except block that matches the exception.

This exception and except statement suffices in most use cases, however, there is a need for a feature that can handle multiple exceptions at a time and a feature that can trigger multiple different exceptions. These features will be particularly useful in asynchronous programming.

ExceptionGroup and the Except* Statement

PEP 654 specification introduces the ExceptionGroup and the except * statement in the Python 3.11 version.

The ExceptionGroup

ExceptionGroup allows you to raise multiple exceptions at the same time.

Like regular exceptions, they are a subclass of the Exception class:

print(issubclass(ExceptionGroup, Exception))

Output:

true

And you can raise them with the raise statement:

raise ExceptionGroup("exception groups", [ValueError(1), TypeError(2)]) 

You can also handle them with the try and except block (you should avoid using except with them as you will see later):

try:
    raise ExceptionGroup("An exception group", [ValueError(), TypeError(1)])
except ExceptionGroup:
    print("I caught an exception group")

Output:

I caught an exception group

Unlike regular exceptions, they take two arguments; the description of the error group and a sequence of exceptions (that needs to be raised at the same time). This sequence can include one or more exceptions of the same or different class, or even an exception group. The sequence cannot be empty though:

ExceptionGroup("An exception group", [ValueError("a value error"), \
SyntaxError("a syntax error")])

If the sequence is empty, you will get the following output:

print(ExceptionGroup("An exception group", []))

Output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: second argument (exceptions) must be a non-empty sequence

The ExceptionGroup have an exceptions attribute that returns a tuple of the exception class in the sequence of exceptions:

print(ExceptionGroup("An exception group", [ValueError("a value error"), \
SyntaxError("a syntax error")]).exceptions)

Output:

(ValueError('a value error'), SyntaxError('a syntax error'))

When you raise them, the traceback shows a hierarchical structure of the exceptions in the group:

raise ExceptionGroup("An exception group", \
[ValueError("a value error"), SyntaxError("a syntax error")])

Output:

  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  | ExceptionGroup: An exception group (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | ValueError: a value error
    +---------------- 2 ----------------
    | SyntaxError: a syntax error
    +------------------------------------

You cannot single out an exception from the ExceptionGroup with the except syntax:

try:
     raise ExceptionGroup("An exception group", [ValueError(), \
     TypeError(1)])
except TypeError:
     print("I am handling the TypeError in the exception group")

Output:

  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 2, in <module>
  | ExceptionGroup: An exception group (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | ValueError
    +---------------- 2 ----------------
    | TypeError: 1
    +------------------------------------

This shows the traceback even though the ExceptionGroup contains the TypeError you handled with the except statement. This is where the except * statement comes in.

The except * Statement

The except * syntax lets you filter and handle a specific exception class from the ExceptionGroup. This removes the exception class from the list of exceptions in the ExceptionGroup.

Unlike the except statement that only executes the first exception block containing the matching exception type, Python checks all the except * blocks even after there is an exception class in the exception group that has matched.

try:
    raise ExceptionGroup("An exception group", [ValueError(), TypeError(1)])
except * TypeError:
    print("I am handling a Type error")
except * ValueError:
    print("I am handling a ValueError")

Output:

I am handling a Type error
I am handling a ValueError

The first except * block filters and handles the TypeError from the exception group. This removes the TypeError exception from the exception sequence that you passed to the ExceptionGroup.

The code went on to handle the ValueError even after there was a block that handled the TypeError exception. This behavior differs from that of the ordinary except statement.

When multiple exceptions of the same class are present in the exception block, the except * filters out and handle the exception of the same class in a single block that matches them:

try:
     raise ExceptionGroup("An exception group", [TypeError(1), \
     TypeError(2)])
except * TypeError as eg:
    print("I am handling TypeError")
    print(eg.exceptions)

Output:

I am handling TypeError
(TypeError(1), TypeError(2))

The except * TypeError block handles both the TypeError(1) and TypeError(2) which is indicated by the output of eg.exceptions.

You cannot use the except * and except statements together on the same try block as this raises a SyntaxError as shown below:

try:
    raise ExceptionGroup("An exception group", [ValueError(), \
    TypeError(1), ValueError(2)])
except * ValueError:
    print("This is value error")
    
except TypeError:
   print("This is a type error")

Output:

  File "<stdin>", line 6
    except TypeError:
    ^^^^^^
SyntaxError: cannot have both 'except' and 'except*' on the same 'try'

You can split ExceptionGroups manually on an exception class with the .split() method that is defined in the BaseExceptionGroup class. This returns two exception groups:

• A group that contains the exception that you split on
• A group that contains the other exceptions in the exception group that you split

Here’s an example:

eg = ExceptionGroup("An exception group", \
[ValueError(1), TypeError(2), SyntaxError(3)])
type_error, other_errors = eg.split(TypeError)
print(type_error)
print(other_errors)

Output:

ExceptionGroup('An exception group', [TypeError(2)])
ExceptionGroup('An exception group', [ValueError(1), SyntaxError(3)])

The code snippet above splits the exception group eg by the TypeError exception. This returns two exception groups; the first group contains the TypeError exception and the other group contains the other exceptions in the initial exception group.

For more on exceptions and exception groups, check out the official documentation.

Exceptions With Notes

Exceptions can be initialized with a message that describes the error:

ValueError("This is a message describing this exception")

This message will be sufficient for most use cases. However, there are times when you might want to add extra information to the exception, information that is generally not available when Python raised the exception. For this purpose, a new add_note(note) method and a __notes__ attribute have been added to the BaseException class in the new Python version. This feature was proposed in the PEP 678 specification.

The __notes__ attribute holds a list of notes that you added in the exception class with the .add_notes() method:

try:
    obj = {"key": "value"}
    print(obj["error"])
except KeyError as exc:
    unavailable_info_b4_exc = "This is just new information" 
    exc.add_note(unavailable_info_b4_exc)
    print("The exception note: ", exc.__notes__)
    raise exc

In this example:

• You accessed a key that is not present in the obj dictionary. This raises a KeyError exception.
• You handled the KeyError exception in the except block.
• You retrieved a piece of hypothetical information that was not available before Python raised the exception.
• You printed the value of the .__notes__ attribute.
• You added this information to the exception note using the .add_note() method and you re-raised the exception.

This output the following traceback error:

The exception note: ['This is just new information]
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
  File "<stdin>", line 4, in <module>
KeyError: 'error'
This is just new information

The traceback now includes the note you added to the exception.

Typing Features

Python does not enforce type at runtime, you can define a type of argument and return type for a function and pass an argument or return an argument of a different type. However, the use of type annotation can still be useful for documenting and improving the readability of your code.

You can use a library like mypy to enforce the type annotation. That will ensure the parameters and the return type conform to the type you specified.

This type annotation feature is aided by the typing module.

The example below shows the signature for a function that returns a boolean value and whose value_1 parameter accepts an integer and the value_2 parameter accepts a string datatype.

def example(value_1:int, value_2:str) -> bool:
   ...

Python also supports the type annotation of generics or containers that can be parameterized. These generics include list, set, tuple, and dict. You can specify the types of parameters these generics accept. This support is denoted by the subscription syntax as shown below:

from typing import Set, List, Tuple

def example(value_1:List[int], value_2:Set[str]) -> Tuple[bool]:
    ...

The value_1 parameter above takes a list of integers, the value_2 parameter takes a set of strings and the function returns a tuple of boolean values.

For generic containers that accept any type of element, the typing module has a factory that can be used to parameterize these generic containers to denote that they accept a type of element. This factory is called the TypeVar class.

In the example below, the T object created from the TypeVar factory denotes a type of element. The function signature denotes that the returned value is consistent with the elements held by the List:

from typing import TypeVar, List

T = TypeVar("T")
def example(value_1: List[T]) -> T:
   ...

This TypeVar only binds to a single type. For instance, if the function above is called with a list of integers, then T is bound to integers.

The new Python version introduces the following changes and upgrades to this type annotation feature:

Variadic Generics

The new Python version introduces a new TypeVarTuple that allows you to parameterize a container type class with an arbitrary number of types of elements as opposed to the TypeVar which allows a single type of elements.

Consider the following example:

from typing import TypeVarTuple, TypeVar, Tuple

TS = TypeVarTuple("TS")
T = TypeVar("T")
def example(value_1:Tuple[T, *TS]):
   ...

The example above shows a function whose value_1 parameter takes in a tuple of arbitrary types of elements.

The above function is called as shown below:

example(value_1=(1, 'a number', 3.0))

The value_1 parameter has a type of (int, str, float). T is bound to int and TS is bound to (str, float).

The TypeVarTuple has to be unpacked with the * syntax because it acts like an arbitrary number of type variables wrapped in a tuple.

If you like to read more about the Variadic Generics, you can check it out in the PEP 646 specification.

TypedDict

TypedDict defines a blueprint that verifies the structure of mapping data types like the Python dictionary.

You can define a blueprint to match the structure of a Python dictionary object as shown below:

from typing import TypedDict

class ArticleType(TypedDict):
   article_id: int
   title: str
   rating: float

This creates the blueprint for a dictionary whose article_id is expected to be an integer, title, a string and the rating, a float data type.

The ArticleType class can be used as shown below:

article_1: ArticleType = {
   "article_id": 23,
   "title": "Introducing the new features in Python 3.11",
   "rating":4.5
}

Before Python 3.11, TypedDict only allows specifying all or none of the dictionary keys as required by passing a total argument to the class that inherits from the TypedDictclass. Setting total to True means all keys are required and False means none:

class ArticleType(TypedDict, total=True):
    …

Python 3.11 introduces Required and NotRequired keywords in the typing module to give you control over which key is required and which is not.

The above type class can now be redefined as shown below:

from typing import TypedDict, Required, NotRequired

class ArticleType(TypedDict):
   article_id: Required[int]
   title: NotRequired[str]
   rating: float

This specifies the article_id as required and the title as not. A key that doesn’t specify this is required by default (unless the total parameter is set to False).

However, none of this type annotation feature is enforced by the Python interpreter. You can still create an article dictionary without specifying the required field.

You can check the REC 655 specification for more detail.

The Self Type

The new Python version also introduces a Self type that can be used to annotate a class method that returns an instance of itself as shown below:

from typing import Self

class Article:
   def a_method_that_returns_an_instance(self) -> Self:
       ...

The TOML File Format and the tomllib Python Library

TOML (Tom’s Obvious Minimal Language), is a new file format that is mostly used in configuration files. It is named after its creator, Tom Preston-Werner. It was created with the main aim that it can be easy to map to a suitable data structure like a dictionary in Python.

The following shows a configuration TOML file that defines the host and port for a database, the frontend and the backend server:

[database]
enabled = true
ports = [ 5432 ]
host = "127.0.0.1"

[servers]

[servers.frontend]
host = "12.3.55.1"
port = [ 3000 ]
role = "frontend"

[servers.backend]
host = "12.4.55.3"
ports = [ 8000 ]
role = "backend"

The new Python 3.11 introduces the tomllib library that you can use to work with this new file format.

You can convert the above configuration into a Python data type as shown below:

import tomllib

toml_string = """
   [database]
enabled = true
ports = [ 5432 ]
host = "127.0.0.1"

[servers]

[servers.frontend]
host = "12.3.55.1"
port = [ 3000 ]
role = "frontend"

[servers.backend]
host = "12.4.55.3"
ports = [ 8000 ]
role = "backend"
"""
data = tomllib.loads(toml_string)
print(data)

The code above wraps a toml configuration as a multiline string and loads the string with the loads() function.

The code above gives the following output:

{'database': {'enabled': True, 'ports': [5432], 'host': '127.0.0.1'}, \
'servers': {'frontend': {'host': '12.3.55.1', 'port': [3000], \
'role': 'frontend'}, 'backend': {'host': '12.4.55.3', 'ports': [8000], \
'role': 'backend'}}}

The file structure is converted into a hierarchical Python dictionary with each block in the configuration holding an inner dictionary of the configuration defined in each block.

The library also has a .load(file) function that can convert a toml file into a python dictionary as shown below:

import tomllib

with open("earthly.toml", "rb") as file:
    data = tomllib.load(file)

To read more about the new library, you can check the official documentation.

Performance Boost

Python speed performance has been the language’s subject of derision for a very long time despite being a great programming language used by a lot of industries. The Python developers are finally proffering a solution to this relatively slow speed.

According to the documentation, CPython 3.11 (the implementation of the Python programming language) is on average 25% faster than CPython 3.10 when measured with the pyperformance benchmark suite, and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup could be up to 10-60% faster.

To verify this, let us profile the execution of joining the numbers between 1 to 100 with a - with the Python timeit module in the command line:

Python 3.11:

$ python3.11 -m timeit '"-".join(str(n) for n in range(100))'

Output:

20000 loops, best of 5: 12.1 usec per loop

That executes the snippet 20000 times and have a best speed of 12.1 microseconds per loop

Python 3.10:

$ python3.10 -m timeit '"-".join(str(n) for n in range(100))'

Output:

20000 loops, best of 5: 16.4 usec per loop

This executes the same snippet 2000 times a best speed of 16.4 microseconds per loop

That’s a difference of 4.3 microseconds which is very noticeable!

You don’t have to change your code or write your code in a specific way to experience this improvement in speed. Just write Pythonic code that follows common best practices, and CPython does the heavy lifting.

It is, however, imperative to point out that certain code won’t have noticeable benefits like code that performs I/O operations or already does most of its computation in a C extension library like NumPy. This improved performance currently benefits pure-Python workloads the most as specified in the documentation.

Conclusion

In this tutorial, we covered some cool new features in Python 3.11 like annotated traceback, the except * syntax, better typing for collections, tomllib, and CPython 3.11’s performance boost. However, this isn’t all, check out the full feature list in the Python documentation.

As you explore these new features and continue to build with Python, consider making your builds consistent and efficient with Earthly. Especially if you’re working with Python builds, Earthly could be a game-changer.

Happy coding!