Python generators and reduce

Question

I am working on a Python3 tornado web server with asynchronous coroutines for GET requests, using the @gen.coroutine decorator. I want to use this function from a library:

@gen.coroutine
def foo(x):
    yield do_something(x)

which is simple enough:

@gen.coroutine
def get(self):
    x = self.some_parameter
    yield response(foo(x))

Now assume there are multiple functions foo1, foo2, etc. of the same type. I want to do something like ...foo3(foo2(foo1(x).result()).result())... and yield that instead of just response(foo(x)) in the get method.

I thought this would be easy with reduce and the result method. However, because of how tornado works, I cannot force the foos to return something with the result method. This means that yield reduce(...) gives an error: "DummyFuture does not support blocking for results". From other answers on SO and elsewhere, I know I will have to use IOLoop or something, which I didn't really understand, and...

...my question is, how can I avoid evaluating all the foos and yield that unevaluated chunk from the get method?

Edit: This is not a duplicate of this question because I want to: 1. nest a lot of functions and 2. try not to evaluate immediately.

Possible duplicate of [Can't call result() on futures in tornado](http://stackoverflow.com/questions/31172272/cant-call-result-on-futures-in-tornado) — Natecat, Jan 04 '17 at 02:54
If you mean return a function(-like) object that simulates that nesting, then [this](http://stackoverflow.com/questions/15331726/how-does-the-functools-partial-work-in-python) is the proper duplicate — Natecat, Jan 04 '17 at 02:57

score 1 · Accepted Answer · answered Jan 04 '17 at 13:25

In Tornado, you must yield a Future inside a coroutine in order to get a result. Review Tornado's coroutine guide.

You could write a reducer that is a coroutine. It runs each coroutine to get a Future, calls yield with the Future to get a result, then runs the next coroutine on that result:

from tornado.ioloop import IOLoop
from tornado import gen


@gen.coroutine
def f(x):
    # Just to prove we're really a coroutine.
    yield gen.sleep(1)
    return x * 2


@gen.coroutine
def g(x):
    return x + 1


@gen.coroutine
def h():
    return 10


@gen.coroutine
def coreduce(*funcs):
    # Start by calling last function in list.
    result = yield funcs[-1]()

    # Call remaining functions.
    for func in reversed(funcs[:-1]):
        result = yield func(result)

    return result


# Wrap in lambda to satisfy your requirement, to 
# NOT evaluate immediately.
latent_result = lambda: coreduce(f, g, h)
final_result = IOLoop.current().run_sync(latent_result)
print(final_result)

Thank you, this makes sense. Could you please explain what happens inside the coreduce function with the multiple yield and return statements? — shardulc, Jan 04 '17 at 17:25
Each "yield" waits for the called coroutine to complete, and resolves the Future it returns into a result. The "return" statement returns a value from the coroutine. Whatever a coroutine returns becomes the value of its Future. — A. Jesse Jiryu Davis, Jan 04 '17 at 19:06
"Whatever a coroutine returns becomes the value of its Future.": this is the most helpful statement I've heard so far about coroutines. Thank you very much! — shardulc, Jan 04 '17 at 19:11

Python generators and reduce

1 Answers1