What's new in Python 3.11?

1. Better error messages

When printing tracebacks, the interpreter will now point to the exact expression that caused the error, instead of just the line.

Example:

Traceback (most recent call last):
  File "distance.py", line 11, in <module>
    print(manhattan_distance(p1, p2))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "distance.py", line 6, in manhattan_distance
    return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
                           ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'x'

As well as complex arithmetic expressions:

Traceback (most recent call last):
  File "calculation.py", line 54, in <module>
    result = (x / y / z) * (a / b / c)
              ~~~~~~^~~
ZeroDivisionError: division by zero

2. Built in toml support

The standard library now has built-in support for reading TOML files, using the tomllib module:

import tomllib

with open('.deepsource.toml', 'rb') as f:
    data = tomllib.load(f)

tomllib is actually based on an open source TOML parsing library called tomli. And currently, only reading TOML files is supported. If you need to write data to a TOML file instead, consider using the tomli-w package.

3. asyncio Task Groups

When doing asynchronous programming, you often run into situations where you have to trigger many tasks to run concurrently, and then take some action when they are completed.

For example, downloading a bunch of images in parallel, and then bundling them to a zip file at the end.

To do that, you need to collect tasks and pass them to asyncio.gather.
Here’s a simple example of parallelly running tasks with the gather function:

import asyncio

async def simulate_flight(city, departure_time, duration):
    await asyncio.sleep(departure_time)
    print(f"Flight for {city} departing at {departure_time}PM")

    await asyncio.sleep(duration)
    print(f"Flight for {city} arrived.")


flight_schedule = {
    'New Delhi': [3, 2],
    'Mumbai': [7, 4],
    'Kolkata': [1, 9],
}

async def main():
    tasks = []
    for city, (departure_time, duration) in flight_schedule.items():
        tasks.append(simulate_flight(city, departure_time, duration))

    await asyncio.gather(*tasks)
    print("Simulations done.")

asyncio.run(main())

Output:

Flight for Kolkata departing at 1PM
Flight for New Delhi departing at 3PM
Flight for New Delhi arrived.
Flight for Mumbai departing at 7PM
Flight for Kolkata arrived.
Flight for Mumbai arrived.
Simulations done.

But having to maintain a list of the tasks yourself to be able to await them is a bit clunky. So now a new python3.11 API is added to asyncio called Task Groups:

import asyncio

async def simulate_flight(city, departure_time, duration):
    await asyncio.sleep(departure_time)
    print(f"Flight for {city} departing at {departure_time}PM")

    await asyncio.sleep(duration)
    print(f"Flight for {city} arrived.")


flight_schedule = {
    'New Delhi': [3, 2],
    'Mumbai': [7, 4],
    'Kolkata': [1, 9],
}

async def main():
    async with asyncio.TaskGroup() as tg:
        for city, (departure_time, duration) in flight_schedule.items():
            tg.create_task(simulate_flight(city, departure_time, duration))

    print("Simulations done.")

asyncio.run(main())

When the asyncio.TaskGroup() context manager exits, it ensures that all the tasks created inside it have finished running.

4. Typing improvements

The typing module saw a lot of interesting updates this release. Here are some of the most exciting ones:

Variadic generics

Support for variadic generics has been added to the typing module in Python 3.11.

What that means is that, now you can define generic types that can take an arbitrary number of types in them. It is useful for defining generic methods for multi-dimensional data.

Example:

from typing import Generic
from typing_extensions import TypeVarTuple

Shape = TypeVarTuple('Shape')

class Array(Generic[*Shape]):
    ...

# holds 1 dimensional data, like a regular list
items: Array[int] = Array()

# holds 3 dimensional data, for example, X axis, Y axis and value
market_prices: Array[int, int, float] = Array()

# This function takes in an `Array` of any shape, and returns the same shape
def double(array: Array[Unpack[Shape]]) -> Array[Unpack[Shape]]:
    ...

# This function takes an N+2 dimensional array and reduces it to an N dimensional one
def get_values(array: Array[int, int, *Shape]) -> Array[*Shape]:
    ...

# For example:
vector_space: Array[int, int, complex] = Array()
reveal_type(get_values(vector_space))  # revealed type is Array[complex]

The new Generic[*Shape] syntax is only supported in Python 3.11. To use this feature in Python 3.10 and below, you can use the typing.Unpack builtin instead: Generic[Unpack[Shape]]

5. Self type

Previously, if you had to define a class method that returned an object of the class itself, adding types for it was a bit weird, it would look something like this:

from typing import TypeVar

T = TypeVar('T', bound=type)

class Circle:
    def __init__(self, radius: int) -> None:
        self.radius = radius

    @classmethod
    def from_diameter(cls: T, diameter) -> T:
        circle = cls(radius=diameter/2)
        return circle

To be able to say that a method returns the same type as the class itself, you had to define a TypeVar, and say that the method returns the same type T as the current class itself.

But with the Self type, none of that is needed:

from typing import Self

class Circle:
    def __init__(self, radius: int) -> None:
        self.radius = radius

    @classmethod
    def from_diameter(cls, diameter) -> Self:
        circle = cls(radius=diameter/2)
        return circle

6. Required[] and NotRequired[]

TypedDict is really useful to add type information to a codebase that uses dictionaries heavily to store data. Here’s how you can use them:

from typing import TypedDict

class User(TypedDict):
    name: str
    age: int

user : User = {'name': "Ram", 'age': 29}
reveal_type(user['age'])  # revealed type is 'int'

However, TypedDicts had a limitation, where you could not have optional parameters inside a dictionary, kind of like default parameters inside function definitions.

For example, you can do this with a NamedTuple:

from typing import NamedTuple

class User(NamedTuple):
    name: str
    age: int
    married: bool = False

ram = User(name='Ram', age=29, married=True)
shyam = User(name='Shyam', age=22)  # 'married' is False by default

This was not possible with a TypedDict (at least without defining multiple of these TypedDict types). But now, you can mark any field as NotRequired, to signal that it is okay for the dictionary to not have that field:

from typing import TypedDict, NotRequired

class User(TypedDict):
    name: str
    age: int
    married: NotRequired[bool]

ram: User = {'name': 'Ram', 'age': 29, 'married': True}
shyam : User = {'name': 'Shyam', 'age': 22}  # 'married' is not required

NotRequired is most useful when most fields in your dictionary are required, having a few not required fields. But, for the opposite case, you can tell TypedDict to treat every single field as not required by default, and then use Required to mark actually required fields.

from typing import TypedDict, Required

# `total=False` means all fields are not required by default
class User(TypedDict, total=False):
    name: Required[str]
    age: Required[int]
    married: bool  # now this is optional

ram: User = {'name': 'Ram', 'age': 29, 'married': True}
shyam : User = {'name': 'Shyam', 'age': 22}  # 'married' is not required

7. contextlib.chdir

contextlib has a small addition to it, which is a context manager called chdir. All it does is change the current working directory to the specified directory inside the context manager, and set it back to what it was before when it exits.

One potential usecase can be to redirect where you write the logs to:

import os

def write_logs(logs):
    with open('output.log', 'w') as file:
        file.write(logs)


def main():
    print("Scanning files...")
    files = os.listdir(os.curdir)  # lists files in current directory
    logs = do_scan(files)

    print("Writing logs to /tmp...")
    with contextlib.chdir('/tmp'):
        write_logs(logs)

    print("Deleting files...")
    files = os.listdir(os.curdir)
    do_delete(files)

This way, you don’t have to worry about changing and reverting the current directory manually, the context manager will do it for you.