Python List Comprehension and Generator Expression Toolbox

Python List comprehension, sometimes called listcomps, and generator expression, (genexps), were a notation inspired by the programming language, Haskell. Their aim is to make code more compact, faster, and optimized provided the author does not make the code unreadable. Many a programmer has found these two notations extremely useful when they want to write pythonic code.

 

To give you an idea behind the inspiration of python generator expressions and list comprehensions along with the syntax for forming them, let’s take a python for loop that iterates through a list of items and appends those items to another list based on a Boolean expression.

The for loop above iterates through a list where fruits are weighted 1 or 2 and placed in a tuple. It then filters all fruits that have a weight of 1 and appends them to the weighted list. To introduce you to the syntax of list comprehensions and generator expressions, we will rewrite the code using list comprehension:

fruits = [(1, 'mango'), (2, 'apple'), (1, 'orange'), (1, 'pineapple'), (2, 'melon'), (1, 'banana')]
weighted_list = [ item[1] for item in fruits if item[0] == 1]
print(weighted_list)

You can run it in the embedded interpreter here:

Compare the two outputs and you will see that they generate the same lists. So, now that you have seen a live demonstration of how a python list comprehension is written, let me explain the syntax of python list comprehensions and generator expressions.

Syntax of python list comprehensions and generator expressions.

The basic syntax for python list comprehension is:

[ expression for item in iterable if condition ]

The basic syntax for generator expression is also:

( expression for item in iterable if condition )

It consists of a for statement which could be followed by an optional if statement and then an expression is returned. Notice that they both have the same syntax. The only difference is that python list comprehensions are surrounded by square brackets while python generator expressions are surrounded by parenthesis. Also, what list comprehension does is to return the expressions as a list while generator expression returns an iterator. So, notice this difference between what they both return because it is very important.

If you want a refresher on what an iterator is or what iterables are, just click on the links.

So, having that syntax, let us show examples of common operations you can use with list comprehensions and generator expressions.

Common operations of list comprehensions and generator expressions.

Two common operations that these two notations perform are: (1) To perform some operation on every element of an iterable. (2) Selecting a subset of elements of an iterable that meets some condition.

Most of the operations you will perform with list comprehensions or generator expressions will fall under one of these two broad categories.

1. Performing some operation on every element of an iterable.

Let’s demonstrate this with python list comprehension examples and generator expression examples.

Suppose you want to add up all the elements of a range of numbers as they are produced. You could most probably use a generator expression for a compact and optimized code. Here is how:

sum_of_num = sum(x for x in range(1, 21))
print(sum_of_num)

With the code above I just added all the numbers from 1 to 20, and the output was 210. Note that since the python generator expression produces an iterator, sum function takes each element of the iterator and adds them together. When you have an iterator that needs operations on their elements, just think of generator expressions.

We could also perform operations on elements of an iterable using list comprehension.

sum_of_num = sum([ x for x in range(1, 21)])
print(sum_of_num)

Notice that I enclosed the list comprehension inside the sum function. This is because sum function can also take an iterable. A list is what the list comprehension produces which is an iterable. Like the generator expression, the list comprehension produced the sum of num as 210. I want you to study both syntax very well and make sure you understand what I did. Now, for further clarification, let me show you the lengthy for loop that the above codes replaced that took longer lines.

summed = []
for x in range(1, 21):
    summed.append(x)
print(sum(summed))

You can see that the python list comprehension and generator expression look more pythonic. One liners are so beautiful. Just compare them and see for yourself.

2. Selecting a subset of elements of an iterable that meets a condition.

Sometimes we want to select elements of an iterable based on a condition being False or True. List comprehensions are usually the handy tool for that job. When I see problems like this, I chose list comprehensions because generator expressions being iterators usually need a function to help bring out the elements. But I will show you how to do this action using both.

For example, suppose we have some numbers and we want to select only the even numbers in the list based on whether the numbers are divisible by 2 without remainder. This is how list comprehension could be written for it.

Run the code above and see for yourself and study the syntax of the list comprehension. You should see a print out of the even numbers selected as a list. Yes, that’s what a list comprehension produces – a list. Now we could do this with a generator expression but because we don’t have a function acting on the elements selected, it doesn’t look that too elegant. We will take note of the fact that the generator expression produces an iterator, so based on the definition of iterators which you remember implements the __next__() special method, we would use the next() method to bring out all the items selected from the range. But since they are ten in number, we would have to call next() ten times.

You don’t expect me to call the next() method 10 times, do you? So you see, for occasions like this when you just need the numbers without doing any operation on them, a list comprehension would suffice.

So, we have our two common broad operations where list comprehensions and generator expressions are usually used.

So, what are the differences between the list comprehension and generator expression?

Differences between list comprehension and generator expression.

The first difference is that while list comprehension will produce a list, a generator expression will produce an iterator. And remember from the post on iterators, they don’t need to materialize all their values at once unless you need those values.

The next difference is that if you are dealing with iterables or iterators that return an infinite stream or a very large amount of data, list comprehensions can compromise your memory. List comprehensions are not memory friendly in this instance because while creating the list they have to use a large amount of space for large data. So, these is where generator expression trumps list expressions because they are more memory friendly, giving you data only when you need them.

As I explained above, you surround python list comprehensions with square brackets while python generator expressions are surrounded with parenthesis. Just wanted to repeat it again. Same syntax but different environment.

As I showed you above, when you just want to select items, it might be better to use a list comprehension but when you want a function to act on the items themselves, a generator expression works better.

Python List comprehensions and generator expressions support nesting

There are times when you have a nested loop. Python list comprehensions and generator expressions also support nesting. Any level of nesting. But be careful not to nest too deeply that the code becomes unreadable.

Let’s show some nesting examples.

On the code above, I wanted to create of list of names to fruits tuples. It is recommended that if a list comprehension will output a tuple, you should surround the tuple with parenthesis as I did above. This is to prevent ambiguity in your code. This just shows that nesting is possible in python list comprehensions and generator expressions.

Nesting can also involve the if statement but be careful that they are well arranged.

So, that is the guide to list comprehensions and generator expressions. Use them with responsibility.

Happy pythoning.