python function call with/without list comprehension

Question

I have below two functions:

def foo(n=50000):
    return sum(i*i for i in range(n))  # just called sum() directly without 

def bar(n=50000):
    return sum([i*i for i in range(n)])  # passed constructed list to sum()

I was hoping that foo will run faster then bar but I have checked in ipython with %%timeit that foo is taking slightly longer then bar

In [2]: %%timeit
   ...: foo(50000)
   ...: 
100 loops, best of 3: 4.22 ms per loop

In [3]: %%timeit
   ...: bar(50000)
   ...: 
100 loops, best of 3: 3.45 ms per loop
In [4]: %%timeit
   ...: foo(10000000)
   ...: 
1 loops, best of 3: 1.02 s per loop

In [5]: %%timeit
   ...: bar(10000000)
   ...: 
1 loops, best of 3: 869 ms per loop

The difference increases as I increase value of n hence I tried to check function with dis.dis(foo) and dis.dis(bar) but it was identical.

So what would be the cause of such time difference between both methods?

Question aside, remember the main reason using a generator expression is preferred in these cases is for improved memory usage rather than speed optimisation — Chris_Rands, Apr 11 '18 at 12:23
I could've sworn we had a couple of questions about this phenomenon already, but I can't find any... — Aran-Fey, Apr 11 '18 at 12:35

Chen A. · Accepted Answer · 2018-04-11T12:56:40.893

6

There are plenty of great answers about generators, so I won't elaborate on that.

Generators keep state. They are slightly slower if you do very fast operations (like using sum, but if you use an I/O command there won't be much difference). The upside for generators is that they don't load all the items to memory in advance, where lists does.

This is what happens when you iterate a list (in very high-level):

You load all items of the list to memory
Asking for the next element just gives you the pointer to that object

Compare that to a generator:

You don't have all the items on memory. Just one item at a time.
Asking for the next element resumes the generator object, running the code until it reaches the yield statement.
Then it yields the object's address in memory so you can access it.

This extra step in the middle, is the diff in your tests.

So, generators are used commonly where you deal with huge amount of data that needs to be loaded on to memory. (There are more use-cases for generators ofcourse, like coroutines)

Do an expirement with huge files and a for loop printing the lines. At some point you will get out of memory exception when using lists. Then try using generators, they won't go out of memory..

edited Apr 11 '18 at 12:56

answered Apr 11 '18 at 12:37

Chen A.

10,140
3
42
61

This is contradictory because a list-comprehension: `[...]` is stated in the documentation to be basically syntactic sugar for `list()` called on a generator. So using a list-comp still has to iterate over the generator asking for the next element, running the code till the yield etc. it just does this at declaration and stores the results in memory rather than on the fly. So I am still personally confused why it is faster to do this at first rather than during the `sum()` call imo your answer doesn't address this you are focusing on memory. – Joe Iddon Apr 11 '18 at 15:32
`list()` builds the data structure first, and loads it to memory. then forwards a list data structure to `sum`. *The list is already built and available*. Where generators yields just one item at a time, and retrieving the next element takes longer as it is executes more code (until next yield) than just getting a pointer back. – Chen A. Apr 11 '18 at 16:02
Yes, I agree that is how it works, but don't see how that would mean lists are faster. The added time of creating the list and loading into memory must cancel out the time saved using a pointer to access the entries? – Joe Iddon Apr 11 '18 at 16:49
Not entirely correct. Creating a list out of range is quick, and you do it once and load it to memory. Then when *iteration* starts it accesses list[index]. All items are accessed in *O(1)* while with generators, accessing an item is a different story - you actually executes code, and yield *one* object. Then again on next iteration, and again.. so yielding an element (in the OP scenario) is more *expensive* in terms of time than just accessing a *ready* list. Also, I'm pretty sure there are some optimizations when generating a list. – Chen A. Apr 11 '18 at 17:33
@joe-iddon Are you possibly referring to [this](https://docs.python.org/3/whatsnew/3.0.html#changed-syntax)? Note that it states that the *semantics* changed so that they are closer to `list()` (like generator expressions they no longer leak the loop control variable, for example), but the implementation is still different: https://stackoverflow.com/questions/30096351/are-list-comprehensions-syntactic-sugar-for-listgenerator-expression-in-pyth. In short your statement that a list-comp still has to iterate over a generator is wrong. – Ilja Everilä Apr 11 '18 at 17:52
@IljaEverilä Yes, I had forgotten about that great answer because I remember looking at this before. I agree I was wrong to say that a list-comp simply iterates over a *generator*, but we can see from the `dis` output that they are roughly similar operations (the difference being the append vs extend). – Joe Iddon Apr 11 '18 at 18:11
@Chen A. The last explanation above in the comment is a bit problematic. I don't think it is right to compare `yield` with list accessing. For list comprehension, the same code for generating the items are executed during the list construction phase and that time is **included** in the measurement in the provided code. So that part should not make so much difference in terms of the **total** execution time as it seems in your explanation. The primary difference seems to be the `yield` statement vs. list construction in the disassembled code. – GZ0 Aug 11 '19 at 04:28

python function call with/without list comprehension

1 Answers1