35

When programming in python, I now avoid map, lambda and filter by using list comprehensions because it is easier to read and faster in execution. But can reduce be replaced as well?

E.g. an object has an operator union() that works on another object, a1.union(a2), and gives a 3rd object of same type.

I have a list of objects:

L = [a1, a2, a3, ...]

How to have the union() of all these objects with list comprehensions, the equivalent of:

result = reduce(lambda a, b :a.union(b), L[1:], L[0])
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Eric H.
  • 2,152
  • 4
  • 22
  • 34
  • 1
    In some cases: no. But depends. Please provide a specific query that you have in mind – sshashank124 Jun 25 '14 at 13:44
  • 1
    @sshashank124 - any examples? – mhawke Jun 25 '14 at 13:53
  • 1
    Set unions are a bad example, because you can simply do `result = set().union(*L)`, which has the bonus of working even if L is an empty list. At any rate, `lambda a, b :a.union(b)` can be written more concisely as `set.union`, since in python `obj.method(args)` is the same as `cls.method(obj, args)` – Eric Mar 11 '16 at 07:07
  • Guido says to use a for loop instead of reduce. He's not a fan of FP constructs. – Didier A. Nov 04 '16 at 20:32

5 Answers5

21

It is no secret that reduce is not among the favored functions of the Pythonistas.

Generically, reduce is a left fold on a list

It is conceptually easy to write a fold in Python that will fold left or right on a iterable:

def fold(func, iterable, initial=None, reverse=False):
    x=initial
    if reverse:
        iterable=reversed(iterable)
    for e in iterable:
        x=func(x,e) if x is not None else e
    return x

Without some atrocious hack, this cannot be replicated in a comprehension because there is not accumulator type function in a comprehension.

Just use reduce -- or write one that makes more sense to you.

dawg
  • 98,345
  • 23
  • 131
  • 206
  • Hi dawg, do you think this still holds true that you can't recreate reduce in a list comprehension in 2023. Great answer regardless – Topde May 25 '23 at 18:25
  • @Topde: Since `reduce` produces a single value and a comprehension produces a list (or set or dict) then it is still true that you have to have some syntax in addition to the comprehension. You *can* generate all the values that get reduced, but that is not really the same thing... – dawg May 25 '23 at 21:06
  • True thanks for responding, I guess we'll keep using reduce for a while then! in my case I've got a function that will check if any value in the input columns is null def is_null(*cols): """Returns true if there's at least one null across all columns.""" return reduce(Column.__or__, [F.col(c).isNull() for c in cols]) the input number of columns can vary so this is a dynamic way of handling that, you can think of this as any('a','b', null) – Topde May 26 '23 at 10:15
8

Since a list comprehension definitionally generates another list, you can't use it to generate a single value. The aren't for that. (Well... there is this nasty trick that uses a leaked implementation detail in old versions of python that can do it. I'm not even going to copy the example code here. Don't do this.)

If you're worried about the stylistic aspects of reduce() and its ilk, don't be. Name your reductions and you'll be fine. So while:

all_union = reduce(lambda a, b: a.union(b), L[1:], L[0])

isn't great, this:

from functools import reduce

def full_union(input):
    """ Compute the union of a list of sets """
    return reduce(set.union, input[1:], input[0])

result = full_union(L)

is pretty clear.

If you're worried about speed, check out the toolz and cytoolz packages, which are 'fast' and 'insanely fast,' respectively. On large datasets, they'll often let you avoid processing your data more than once or loading the whole set in memory at once, in contrast to list comprehensions.

Nate
  • 4,561
  • 2
  • 34
  • 44
  • 2
    To make the `reduce()` expression itself readable, make the first argument not a lambda. For example: `reduce(set.union, )` Sometimes this will require you to define (and therefore name) the operator somewhere outside the call to `reduce`. – Jordan Oct 11 '17 at 00:12
5

A common use of reduce is to flatten a list of lists. You can use a list comprehension instead.

L = [[1, 2, 3], [2, 3, 4], [3, 4, 5]]

with reduce

from functools import reduce  # python 3
flattened = reduce(lambda x, y: x + y, L)

print(flattened)

[1, 2, 3, 2, 3, 4, 3, 4, 5]

with list comp

flattened = [item for sublist in L for item in sublist]

print(flattened)

[1, 2, 3, 2, 3, 4, 3, 4, 5]

If your problem can be solved by operating on the flattened list, this is an effective replacement. Contrast these one-liners for the given example:

all_union = reduce(lambda a, b: set(a).union(set(b)), L)

{1, 2, 3, 4, 5}

all_union = set([item for sublist in L for item in sublist])

{1, 2, 3, 4, 5}
Nicholas Morley
  • 3,910
  • 3
  • 16
  • 14
4

Not really. List comprehensions are more similar to map and filter.

hd1
  • 33,938
  • 5
  • 80
  • 91
bigblind
  • 12,539
  • 14
  • 68
  • 123
0

Not the most readable reduce for unions, but assignment expressions can be used to solve it in a single expression:

In [51]: L = [{1}, {2}, {3}, {4}, {5}]

In [52]: reduce(lambda a, b: a.union(b), L[1:], L[0])
Out[52]: {1, 2, 3, 4, 5}

In [53]: (s := set(), [s := s.union(x) for x in L][-1])[1]
Out[53]: {1, 2, 3, 4, 5}
dawid
  • 663
  • 6
  • 12