Monday, October 2, 2023

Python Asyncio Libraries 2023

Web Frameworks

Libraries to build web applications.

  • FastAPI - A very high performance Python 3.6+ API framework based on type hints. Powered by Starlette and Pydantic.
  • Django - An established, high-level Python web framework with a huge community and ecosystem.
  • Starlette - A lightweight ASGI framework/toolkit for building high performance services.
  • aiohttp - Http client/server for asyncio (PEP-3156).
  • sanic - Python 3.5+ web server that's written to go fast.
  • Quart - An asyncio web microframework with the same API as Flask.
  • autobahn - WebSocket and WAMP supporting asyncio and Twisted, for clients and servers.
  • websockets - A library for building WebSocket servers and clients in Python with a focus on correctness and simplicity.
  • Tornado - Performant web framework and asynchronous networking library.
  • uvicorn - The lightning-fast ASGI server.

Message Queues

Libraries to implement applications using message queues.

  • aioamqp - AMQP implementation using asyncio.
  • pyzmq - Python bindings for ZeroMQ.
  • aiozmq - Alternative Asyncio integration with ZeroMQ.
  • crossbar - Crossbar.io is a networking platform for distributed and microservice applications.
  • asyncio-nats - Client for the NATS messaging system.
  • aiokafka - Client for Apache Kafka.

Database Drivers

Libraries to connect to databases.

  • asyncpg - Fast PostgreSQL Database Client Library for Python/asyncio.
  • asyncpgsa - Asyncpg with sqlalchemy core support.
  • aiopg - Library for accessing a PostgreSQL database.
  • aiomysql - Library for accessing a MySQL database
  • aioodbc - Library for accessing a ODBC databases.
  • motor - The async Python driver for MongoDB.
  • redis-py - Redis Python Client (which includes aioreadis now).
  • aiocouchdb - CouchDB client built on top of aiohttp (asyncio).
  • aioinflux - InfluxDB client built on top of aiohttp.
  • aioes - Asyncio compatible driver for elasticsearch.
  • peewee-async - ORM implementation based on peewee and aiopg.
  • GINO - is a lightweight asynchronous Python ORM based on SQLAlchemy core, with asyncpg dialect.
  • Tortoise ORM - native multi-backend ORM with Django-like API and easy relations management.
  • Databases - Async database access for SQLAlchemy core, with support for PostgreSQL, MySQL, and SQLite.
  • Prisma Client Python - An auto-generated, fully type safe ORM powered by Pydantic and tailored specifically for your schema - supports SQLite, PostgreSQL, MySQL, MongoDB, MariaDB and more.
  • Piccolo - An ORM / query builder which can work in async and sync modes, with a nice admin GUI, and ASGI middleware.
  • Beanie - An async MongoDB ODM built on motor and Pydantic.

Networking

Libraries to communicate in your network.

  • AsyncSSH - Provides an asynchronous client and server implementation of the SSHv2 protocol.
  • aiodns - Simple DNS resolver for asyncio.
  • aioping - Fast asyncio implementation of ICMP (ping) protocol.
  • httpx - asynchronous HTTP client for Python 3 with requests compatible API.

GraphQL

Libraries to build GraphQL servers.

  • Ariadne - Schema-first Python library for implementing GraphQL servers.
  • Tartiflette - Schema-first Python 3.6+ GraphQL engine built on top of libgraphqlparser.
  • Strawberry - Code-first Python 3 GraphQL server with Django, Flask and FastAPI/Starlette support.

Testing

Libraries to test asyncio based applications.

  • aiomock - A python mock library that supports async methods.
  • asynctest - Enhance the standard unittest package with features for testing. asyncio libraries
  • pytest-asyncio - Pytest support for asyncio.
  • aresponses - Asyncio http mocking. Similar to the responses library used for requests.
  • aioresponses - Helper for mock/fake web requests in Python aiohttp package.

Alternative Loops

Alternative asyncio loop implementations.

  • uvloop - Ultra fast implementation of asyncio event loop on top of libuv.

Misc

Other awesome asyncio libraries.

  • aiochan - CSP-style concurrency with channels, select and multiprocessing on top of asyncio.
  • aiocache - Cache manager for different backends.
  • aiofiles - File support for asyncio.
  • aiopath - Asynchronous pathlib for asyncio.
  • aiodebug - A tiny library for monitoring and testing asyncio programs.
  • aiorun - A run() function that handles all the usual boilerplate for startup and graceful shutdown.
  • aiosc - Lightweight Open Sound Control implementation.
  • aioserial - A drop-in replacement of pySerial.
  • aiozipkin - Distributed tracing instrumentation for asyncio with zipkin
  • asgiref - Backend utils for ASGI to WSGI integration, includes sync_to_async and async_to_sync function wrappers.
  • async_property - Python decorator for async properties.
  • ruia - An async web scraping micro-framework based on asyncio.
  • kubernetes_asyncio - Asynchronous client library for Kubernetes.
  • aiomisc - Miscellaneous utils for asyncio.
  • taskiq - Asynchronous distributed task manager (like celery, but async).

Writings

Documentation, blog posts, and other awesome writing about asyncio.

Talks

Recordings of awesome talks about asyncio.

Alternatives to asyncio

Alternative approaches to async programming in Python, some of which attempt to support some compatibility with asyncio, others are not compatible at all.

  • curio - The coroutine concurrency library.
  • trio - Pythonic async I/O for humans and snake people.
    • trio-asyncio - re-implementation of the asyncio mainloop on top of Trio.
  • AnyIO - High level asynchronous concurrency and networking framework that works on top of either trio or asyncio.

Source: https://github.com/timofurrer/awesome-asyncio 

*****

A list of libraries for AsyncIO is available on PyPI with Framework::AsyncIO classifier.

Alternative event loop implementations

  • uvloop: Ultra fast implementation of asyncio event loop on top of libuv. Read more about uvloop here.
  • async-tokio: Asyncio event loop implementation, written in Rust language

Libraries

  • aiofiles: File support for asyncio
  • aiocache: Caching for asyncio with multiple backends (framework agnostic)
  • aiorwlock: Read write lock for asyncio
  • aioutils: Python3 Asyncio Utils, Group (like gevent.pool.Group), Pool (like gevent.pool.Pool), Bag and OrderedBag
  • tasklocals: Task-local variables
  • janus: Thread-safe asyncio-aware queue
  • asyncio_extras: asynchronous generator/context manager support, thread pool integration, async file operations and more
  • paco: Small utility library for coroutine-driven asynchronous programming
  • aioconsole: Asynchronous console and interfaces for asyncio
  • aiostream: Generator-based operators for asynchronous iteration

Unit testing

  • asynctest: Enhance the standard unittest package for testing asyncio libraries
  • pytest-asyncio: Pytest support for asyncio

Protocols implementations

Clients

  • AMQP:
    • aioamqp: AMQP implementation using asyncio (Used by RabbitMQ)
    • QBroker: easy-to-use library based on aioamqp, for pub/sub and RPC
  • AMI: panoramisk, a library to play with Asterisk's protocols: AMI and FastAGI
  • Cassandra: aiocassandra, Simple threaded cassandra wrapper for asyncio
  • CouchDB: aiocouchdb, CouchDB client
  • DNS: aiodns: Async DNS resolver
  • ElasticSearch:
  • Etcd: aioetcd: Coroutine-based etcd client
  • Google Hangouts: hangups: Client for Google Hangouts
  • HTTP: aiohttp.requests: http client with Requests-like API.
  • IRC:
    • irc3, plugable irc client library based on python's asyncio
    • bottom, asyncio-based rfc2812-compliant IRC Client
  • Kafka: aiokafka: Full featured Kafka client for Kafka 9.0.0+
  • memcached: aiomemcache, minimal memcached client
  • MongoDB:
  • MPD: aiompd, Music Player Daemon client for AsyncIO
  • MySQL: aiomysql, MySQL driver
  • ODBC: aioodbc, ODBC client on top of pyodbc, works only with python 3.5+
  • PostgreSQL:
    • aiopg, PostgreSQL client library built on top of psycopg2
    • aiopg8000, A Pure-Python PostgreSQL driver for AsyncIO
    • asyncpg, A fast PostgreSQL Database Client Library for Python/asyncio
  • Prometheus: prometheus_async adds support for asynchronous frameworks to the official Python client for the Prometheus metrics and monitoring system.
  • PySerial: asyncserial: asyncio support for pySerial
  • Redis:
  • Riak: aioriak: Python asyncio client for Riak
  • Slack aioslacker: slacker wrapper for asyncio
  • SIP: aiosip: SIP support for AsyncIO
  • Socks: aiosocks: SOCKS proxy client for asyncio and aiohttp
  • SSH: AsyncSSH: SSH client and server implementation
  • WebSockets: aiohttp.ws_connect: WebSockets client
  • XMPP (Jabber):
    • slixmpp, SleekXMPP (XMPP Library) fork using asyncio, for poezio,
    • aioxmpp, from scratch XMPP library for asyncio (alpha)
  • Zipkin: aiozipkin: Distributed tracing instrumentation for asyncio with zipkin

Servers

  • FastAGI: panoramisk, a library to play with Asterisk's protocols: AMI and FastAGI
  • IRC: irc3d, irc server library based on irc3
  • HTTP: aiohttp: http client and server infrastructure for asyncio
  • SSH: AsyncSSH: SSH client and server implementation
  • SMTP/LMTP: aiosmtpd: Server implementations and testing frameworks for Simple Mail Transport Protocol and Local Mail Transport Protocol.

Web Frameworks

For classical HTTP/1.1 protocol

Looking for WSGI? Read this thread: WSGI implementation compatible with asyncio?.

  • aiohttp.web: a Flask-like API to build quickly HTTP applications, made by the creators of aiohttp
  • aiopyramid: Tools for running pyramid using asyncio
  • aiotraversal: Web framework for big complicated applications based on aiohttp
  • aiowsgi: minimalist wsgi server using asyncio
  • Growler: A web framework built on top of aiohttp
  • interest: event-driven web framework on top of aiohttp/asyncio
  • muffin: A web framework based on Asyncio stack (early alpha)
  • nacho: web framework
  • Pulsar: Event driven concurrent framework for python. With pulsar you can write asynchronous servers performing one or several activities in different threads and/or processes.
  • rainfall: another web framework
  • sanic: A Flask-like Python 3.5+ web server that's written to go fast.
  • Vase: web framework

For WebSockets

ORMs

  • aiopg.sa: support for SQLAlchemy functional SQL layer, based on aiopg
  • aiomysql.sa: support for SQLAlchemy functional SQL layer, based on aiomysql
  • peewee: peewee-async, library providing asynchronous interface powered by asyncio for peewee ORM.
  • GINO: lightweight ORM based on asyncpg and SQLAlchemy core

Integration with other applications

  • API-Hour: Transform easily your AsyncIO server source code (HTTP, SSH, ...) to be multiprocess: it will help to improve the efficency on multi-CPU servers.
  • Gunicorn: Gunicorn has gaiohttp worker built on top of aiohttp library

Applications built with AsyncIO

  • ktcal2: SSH brute forcer tool and library, using AsyncIO of Python 3.4

Run asyncio on top of

  • eventlet: aioeventlet, asyncio API implemented on top of eventlet

Adapters for other event loops

Some people have already written adapters for integrating asyncio with other async I/O frameworks.

Misc

  • aioauth-client: OAuth1/2 support (authorization, resource's loading)
  • aiocron: Crontabs for asyncio
  • aiomas: A library for multi-agent systems and RPC based on asyncio
  • aiotest: test suite to validate an implementation of the asyncio API
  • aioprocessing: A Python 3.3+ library that integrates the multiprocessing module with asyncio
  • blender-asyncio: Asyncio Bridge for Blender Python API
  • ipython-yf: An ipython extension to make it asyncio compatible
  • aiogearman: asyncio gearman support. Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work.
  • aiogear: Yet another gearman protocol implemented in asyncio.
  • Serial port using the serial module, see using serial port in python3 asyncio at Stackoverflow, serial.Serial can be registered with loop.add_reader().
  • async_lru: port of functools.lru_cache to asyncio
  • GoogleScraper: A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, Baidu and others) by using proxies (socks4/5, http proxy) and with many different IP's.
  • aiotasks: A Celery like task manager that distributes Asyncio coroutines.

Filesystem

asyncio does not support asynchronous operations on the filesystem. Even if files are opened with O_NONBLOCK, read and write will block.

Read asynchronous disk I/O (October 2012 by arvid).

The Linux kernel provides asynchronous operations on the filesystem (aio), but it requires a library and it doesn't scale with many concurrent operations. See aio.

The GNU C library (glibc) implements the POSIX aio interface, but it is implemented with threads. See aio(7) manual page.

Recent discussion on the Linux Kernel: Non-blocking buffered file read operations (September 2014).

For now, the workaround is to use aiofiles that uses threads to handle files.

at 5:35 PM
Labels: , , , , ,