[Python asyncio]

Tags: #python #asyncio #helloworld

0. OUTDATED.md

Refer to my blog post here:

https://www.integralist.co.uk/posts/python-asyncio/

1. Python Asyncio.md

The following guide is based on Python 3.8

asyncio is a library to write concurrent code using the async/await syntax. – https://docs.python.org/3.8/library/asyncio.html

The asyncio module provides both high-level and low-level APIs. Library and Framework developers will be expected to use the low-level APIs, while all other users are encouraged to use the high-level APIs.

Event Loop

The core element of all asyncio applications is the ‘event loop’. The event loop is what schedules and runs asynchronous tasks (it also handles network IO operations and the running of subprocesses).

Awaitables

Something is awaitable if it can be used in an await expression.

There are three main types of awaitables:

Coroutines
Tasks
Futures

Note: Futures is a low-level type and so you shouldn’t need to worry about it too much if you’re not a library/framework developer (as you should be using the higher-level abstraction APIs instead).

Coroutines

There are two closely related terms used here:

a coroutine function: an async def function.
a coroutine object: an object returned by calling a coroutine function.

Generator based coroutine functions (e.g. those defined by decorating a function with @asyncio.coroutine) are superseded by the async/await syntax, but will continue to be supported until Python 3.10 – https://docs.python.org/3.8/library/asyncio-task.html#asyncio-generator-based-coro

Tasks

Tasks are used to schedule coroutines concurrently.

All asyncio applications will have (at least) a single ‘main’ entrypoint task that will be scheduled to run immediately on the event loop. This is done using the asyncio.run function (see ‘Running an asyncio program’).

A coroutine function is expected to be passed to asyncio.run, while internally asyncio will check this using the helper function coroutines.iscoroutine. If not a coroutine, then an error is raised, otherwise the coroutine will be passed to loop.run_until_complete. The run_until_complete function expects a Future (see below section) and uses another helper function futures.isfuture to check the type provided.

In older versions of Python you would have used asyncio.ensure_future (now considered to be a low-level API) to schedule a coroutine to be executed on the event loop, but with Python 3.7+ this has been superseded by asyncio.create_task.

Additionally, with Python 3.8, the idea of interacting with the event loop directly (e.g. getting the event loop, creating a task with create_task and then passing it to the event loop) has been replaced with asyncio.run, which abstracts it all away for you (see ‘Running an asyncio program’ to understand what that means).

The following APIs let you see the state of the tasks running on the event loop:

asyncio.current_task
asyncio.all_tasks

Note: for other available methods on a Task object please refer to the documentation.

Futures

A Future is a low-level awaitable object that represents an eventual result of an asynchronous operation.

To use an analogy: it’s like an empty postbox. At some point in the future the postman will arrive and stick a letter into the postbox.

This API exists to enable callback-based code to be used with async/await, while loop.run_in_executor is an example of an asyncio low-level API function that returns a Future (see also some of the APIs listed in Concurrent Functions).

Note: for other available methods on a Future please refer to the documentation.

Running an asyncio program

The high-level API (as per Python 3.7+) is:

import asyncio

async def foo():
    print("Foo!")

async def hello_world():
    await foo()  # waits for `foo()` to complete
    print("Hello World!")

asyncio.run(hello_world())

The .run function always creates a new event loop and closes it at the end. If you were using the lower-level APIs, then this would be something you’d have to handle manually (as demonstrated below).

loop = asyncio.get_event_loop()
loop.run_until_complete(hello_world())
loop.close()

Concurrent Functions

asyncio.gather: takes a sequence of awaitables, returns an aggregate list of successfully awaited values.
asyncio.shield: prevent an awaitable object from being cancelled.
asyncio.wait: wait for a sequence of awaitables, until the given ‘condition’ is met.
asyncio.wait_for: wait for a single awaitable, until the given ‘timeout’ is reached.
asyncio.as_completed: similar to gather but returns Futures that are populated when results are ready.

Note: gather has specific options for handling errors and cancellations. For example, if return_exceptions: False then the first exception raised by one of the awaitables is returned to the caller of gather, where as if set to True then the exceptions are aggregated in the list alonside successful results. If gather() is cancelled, all submitted awaitables (that have not completed yet) are also cancelled.

Deprecated functions

@asyncio.coroutine: removed in Python 3.10
asyncio.sleep: removed in Python 3.10

Note: you’ll find in most of these APIs a loop argument can be provided to enable you to indicate the specific event loop you want to utilize). It seems Python has deprecated this argument in 3.8, and will remove it completely in 3.10.

Older Examples

See gist ‘comments’ section for the following…