1

There is a library that uses blocking requests in its core and I would like to rewrite it into asynchronous version, so could you please advise what would be the best/easiest strategy to do so.

The whole library, after several nested functions, calls one function:

def _send_http_request(self, url, payload, method='post', **kwargs):
# type: (Text, Optional[Text], Text, dict) -> Response

response = request(method=method, url=url, data=payload, **kwargs)

return response

Just putting async in front of it wont work since it is deeply nested in blocking functions. And rewriting everything would be a way too much hassle. I had a look into aiohttp, trio, asks and kinda got lost, which one is better. I know about celery or dask, but I need async.

stkubr
  • 371
  • 1
  • 5
  • 15
  • what do you mean you need async? You need to fire a function call asynchronously and provide some sort of callback ? – ranjith Mar 12 '19 at 20:27

1 Answers1

1

You have several options:

  1. Rewrite _send_http_request to be async (using, for example, aiohttp) and further rewrite all functions that use _send_http_request to be async either. Yes, it's much work to do, but this is how asyncio fundamentally designed.

  2. Wrap only top-level blocking functions (functions with I/O) you need to run asynchronously with run_in_executor as explained here. If you aren't going to make millions of requests you won't see much performance difference with option above since main bottleneck is still I/O. Otherwise threads overhead will be noticeable compared to pure asyncio solution.

  3. Try other solution instead of asyncio. For example, gevent and its monkey-patching. This approach has own pros and cons.

Mikhail Gerasimov
  • 36,989
  • 16
  • 116
  • 159