Python Map Function And Its Components

Very often, we want to apply a function to an iterable without using a for loop. We saw an example in the python reduce function but the python reduce function doesn’t really fit want we want to do because the reduce function successively accumulates the results. We want the function to be applied to each element of the iterable and be stored separately. To do that, we would use a python map. It is just the right function for the job. In this blog post, I will show you how to use a python map and also its subsequent, a python starmap, along with usage.

 

What is a python map.

A python map is just a function that takes iterables and applies a function to the elements of the iterables. If there are more than one iterable, it applies the function to the corresponding elements of the iterable in succession. If the iterables are of different lengths, it stops at the shortest iterable. The result of a python map function is an iterator object. That means to get out the results you have to apply another function to the object. Most times, you would cast it to a list or set.

The syntax of the python map function is map(function, iterable, ...) where function is the function you want to apply to the items of the iterable and iterable is the object which contains the items.

Visit this link if you want to refresh yourself on iterators, and this other one on python iterables. They are important concepts in python. I explained them in depth.

Now, let’s take some examples.

We’ll show examples using a single python iterable and then when more than one python iterable is used. First using a single python iterable.

Supposing we have a list of numbers and we have a function that raises a given number by a power of 3. We could use python map to apply the function to each of the items in the list of numbers.

You will notice from the code above that I used the python map function to apply power_func to each of the items of the num_list in line 5. The first time we printed out the object, items_raised, what we get is a map object. The map object is an iterator. So, to get out the elements in the iterator we cast to a list in line 8 and it then extracted each of the items raised.

Now let’s show an example with more than one iterable. This time, we will use two iterables and add their items together.

What we did above is to provide two iterables, num_list1 and num_list2, to the python map function, and then use the function, adder. What map does is take the items at corresponding indices and pass them to adder which adds them together and then provides the result to the map iterator. Then using list function, we extract each of the elements in the map iterator which is then printed out in line 7.

There is also another case I want you to consider if using more than one iterable and the iterables are not of equal length. What a python map does is that it applies the function to the items of the iterable successively until it comes to the end of the shorter length iterable and then it stops. Let’s take an example, this time letting num_list2 be longer than num_list1.

You can see this time that it stops short at the items at index 4 in both lists and ignores the rest of the items in num_list2 because num_list2 is longer.

One thing you need to know is that the python map function falls short when the items of the iterable is a tuple. This is because map takes each of the items as a single element and was not meant to work with tuples. But not to worry, we have another descendant of map, the python starmap function, that helps us to deal with tuples as elements.

What is the python starmap function?

Just like the python map function, the python starmap function returns an iterator based on the operation of a provided function to the elements of an iterable but this time it works when the elements are tuples. Since you know the basic concepts behind python maps, it also applies to python starmaps. The python starmap function is included with the itertools module. So to use it, you first need to import it from the itertools module.

The syntax for starmap function is itertools.starmap(function, iterable) where function is the function carrying out the operation on each of the elements of the iterable. The elements of the iterable are arranged in tuples.

As an illustration, let’s take an iterable of tuples for example.

As you can see from above, in line 1 we first imported the python starmap function from itertools. Then we defined a function, power, that takes in two arguments and raises the first argument to the power of the second argument at lines 3 and 4. Then at line 7 we used the starmap function to apply the power function to each of the elements of the iterable, this time a list, which are tuples. Python starmap unpacks the tuple when sending them to the power function such that the first item in the tuple becomes bound to x and the second item of the tuple becomes bound to y, and then the function is applied to them such that x is raised to power y and the result is added to the starmap object. Then in line 8 we call list function on the starmap object which is an iterator to extract each of the items in the iterator. Then finally, we print them out.

We can use an iterable with a tuple that has any number of items as long as the function to which they would be applied can accept that number of arguments when the tuple is unpacked by the python starmap function. Let’s take another example of a starmap being used with an iterable that has a tuple of three items.

In the code above, the functioin, is_pythagoras, is based on the Pythagoras rule that the square of two numbers in a triangle is equal to the square of the longest side. What is_pythagoras function does is take three positional arguments and checks for the Pythagoras rule in the arguments. If it obeys the pythagors rule, it returns True but if not it returns False. Then in line 9 we created a list of tuples that is structured with 3 items representing the sides of a triangle, with the third item being the longest side. Then we applied the triangles list and the is_pythagoras function together in the starmap function to check which side obeys the Pythagoras rule. You will notice that line 10 produces a list having either True or False as entries. Then in line 11 to 14, we checked which of the entries in the list has the True value and then printed the corresponding entry from the triangles list of tuples as the tuple obeying the Pythagoras rule.

I hope you enjoyed yourself as I did. These functions show you the powerful abilities of python. Use it to your enjoyment. To keep receiving updates from me on how to use python, you can subscribe to my blog. Thanks for reading.

Happy pythoning.

Python List Comprehension and Generator Expression Toolbox

Python List comprehension, sometimes called listcomps, and generator expression, (genexps), were a notation inspired by the programming language, Haskell. Their aim is to make code more compact, faster, and optimized provided the author does not make the code unreadable. Many a programmer has found these two notations extremely useful when they want to write pythonic code.

 

To give you an idea behind the inspiration of python generator expressions and list comprehensions along with the syntax for forming them, let’s take a python for loop that iterates through a list of items and appends those items to another list based on a Boolean expression.

The for loop above iterates through a list where fruits are weighted 1 or 2 and placed in a tuple. It then filters all fruits that have a weight of 1 and appends them to the weighted list. To introduce you to the syntax of list comprehensions and generator expressions, we will rewrite the code using list comprehension:

fruits = [(1, 'mango'), (2, 'apple'), (1, 'orange'), (1, 'pineapple'), (2, 'melon'), (1, 'banana')]
weighted_list = [ item[1] for item in fruits if item[0] == 1]
print(weighted_list)

You can run it in the embedded interpreter here:

Compare the two outputs and you will see that they generate the same lists. So, now that you have seen a live demonstration of how a python list comprehension is written, let me explain the syntax of python list comprehensions and generator expressions.

Syntax of python list comprehensions and generator expressions.

The basic syntax for python list comprehension is:

[ expression for item in iterable if condition ]

The basic syntax for generator expression is also:

( expression for item in iterable if condition )

It consists of a for statement which could be followed by an optional if statement and then an expression is returned. Notice that they both have the same syntax. The only difference is that python list comprehensions are surrounded by square brackets while python generator expressions are surrounded by parenthesis. Also, what list comprehension does is to return the expressions as a list while generator expression returns an iterator. So, notice this difference between what they both return because it is very important.

If you want a refresher on what an iterator is or what iterables are, just click on the links.

So, having that syntax, let us show examples of common operations you can use with list comprehensions and generator expressions.

Common operations of list comprehensions and generator expressions.

Two common operations that these two notations perform are: (1) To perform some operation on every element of an iterable. (2) Selecting a subset of elements of an iterable that meets some condition.

Most of the operations you will perform with list comprehensions or generator expressions will fall under one of these two broad categories.

1. Performing some operation on every element of an iterable.

Let’s demonstrate this with python list comprehension examples and generator expression examples.

Suppose you want to add up all the elements of a range of numbers as they are produced. You could most probably use a generator expression for a compact and optimized code. Here is how:

sum_of_num = sum(x for x in range(1, 21))
print(sum_of_num)

With the code above I just added all the numbers from 1 to 20, and the output was 210. Note that since the python generator expression produces an iterator, sum function takes each element of the iterator and adds them together. When you have an iterator that needs operations on their elements, just think of generator expressions.

We could also perform operations on elements of an iterable using list comprehension.

sum_of_num = sum([ x for x in range(1, 21)])
print(sum_of_num)

Notice that I enclosed the list comprehension inside the sum function. This is because sum function can also take an iterable. A list is what the list comprehension produces which is an iterable. Like the generator expression, the list comprehension produced the sum of num as 210. I want you to study both syntax very well and make sure you understand what I did. Now, for further clarification, let me show you the lengthy for loop that the above codes replaced that took longer lines.

summed = []
for x in range(1, 21):
    summed.append(x)
print(sum(summed))

You can see that the python list comprehension and generator expression look more pythonic. One liners are so beautiful. Just compare them and see for yourself.

2. Selecting a subset of elements of an iterable that meets a condition.

Sometimes we want to select elements of an iterable based on a condition being False or True. List comprehensions are usually the handy tool for that job. When I see problems like this, I chose list comprehensions because generator expressions being iterators usually need a function to help bring out the elements. But I will show you how to do this action using both.

For example, suppose we have some numbers and we want to select only the even numbers in the list based on whether the numbers are divisible by 2 without remainder. This is how list comprehension could be written for it.

Run the code above and see for yourself and study the syntax of the list comprehension. You should see a print out of the even numbers selected as a list. Yes, that’s what a list comprehension produces – a list. Now we could do this with a generator expression but because we don’t have a function acting on the elements selected, it doesn’t look that too elegant. We will take note of the fact that the generator expression produces an iterator, so based on the definition of iterators which you remember implements the __next__() special method, we would use the next() method to bring out all the items selected from the range. But since they are ten in number, we would have to call next() ten times.

You don’t expect me to call the next() method 10 times, do you? So you see, for occasions like this when you just need the numbers without doing any operation on them, a list comprehension would suffice.

So, we have our two common broad operations where list comprehensions and generator expressions are usually used.

So, what are the differences between the list comprehension and generator expression?

Differences between list comprehension and generator expression.

The first difference is that while list comprehension will produce a list, a generator expression will produce an iterator. And remember from the post on iterators, they don’t need to materialize all their values at once unless you need those values.

The next difference is that if you are dealing with iterables or iterators that return an infinite stream or a very large amount of data, list comprehensions can compromise your memory. List comprehensions are not memory friendly in this instance because while creating the list they have to use a large amount of space for large data. So, these is where generator expression trumps list expressions because they are more memory friendly, giving you data only when you need them.

As I explained above, you surround python list comprehensions with square brackets while python generator expressions are surrounded with parenthesis. Just wanted to repeat it again. Same syntax but different environment.

As I showed you above, when you just want to select items, it might be better to use a list comprehension but when you want a function to act on the items themselves, a generator expression works better.

Python List comprehensions and generator expressions support nesting

There are times when you have a nested loop. Python list comprehensions and generator expressions also support nesting. Any level of nesting. But be careful not to nest too deeply that the code becomes unreadable.

Let’s show some nesting examples.

On the code above, I wanted to create of list of names to fruits tuples. It is recommended that if a list comprehension will output a tuple, you should surround the tuple with parenthesis as I did above. This is to prevent ambiguity in your code. This just shows that nesting is possible in python list comprehensions and generator expressions.

Nesting can also involve the if statement but be careful that they are well arranged.

So, that is the guide to list comprehensions and generator expressions. Use them with responsibility.

Happy pythoning.

5 Python Directory Handling Techniques

Directories and files are crucial to a programmer who wants a resource for his programs. That is why it is necessary after discussing python’s file handling methods, one should also undertake an understanding of python’s directory handling methods or routines. In this post, I will describe 5 routine ways one can handle directories using the methods provided in python such as the python make directory method and get working directory methods.

 

To be able to run any of the commands in this post, you first need to import the os module into your interpreter. To do that you use the code: import os.

Getting the python current directory

The current directory is the directory from which the python interpreter is operating. It depends on how you launch your interpreter or your editor. To know your current working directory is easy. You just need to call the python get current working directory method, os.getcwd(). Here is an example:

import os

working_dir = os.getcwd()
print(working_dir)

The code above will make the current working directory to be printed on your terminal.

You might desire to change your current working directory. Maybe you want to do some experiment on some programs and want to run them on a directory you intend to delete later; I do that all the time. Changing the current working directory is easy with python. You use the python change working directory method, whose syntax is: os.chdir(path). It states that you have to provide a path as an argument for the directory you want to switch to. Path should be based on the path specification of your operating system. It is wise to make path a string in all cases.

An example will suffice.

import os

os.chdir('C:\\Users\\Michael\\Desktop\\')
print(os.getcwd())

Notice above that I double escaped the backslash character. This is because the backslash is a special character. When I run the above code, it changed my working directory to ‘C:\Users\Michael\Desktop\’. Also, I am working on a windows 10 computer in case you are using Unix, Linux or Mac.

Creating New Directories with Python

There are occasions you want to create new directories, or what some call make new directories in python. Python can do this very easily when you use the right methods. There are two methods provided in python: the python make directory method, os.mkdir, and the python make directory recursive method, os.makedirs, which acts recursively by creating more than one directory as long as the directories do not already exist.

The syntax of the os.mkdir method is: os.mkdir(path, mode=0o777, *, dir_fd=None) where path is the name of the directory you want to create. You can leave the other keyword defaults as is because on some systems the mode parameter is just ignored and directory file descriptors, dir_fd, are not implemented.

Supposing we want to create a directory called, new_dir, we could try the following:

import os

try:
    os.mkdir('new_dir')
except FileExistsError:
    print('Directory already exists.')
else:
    print('Directory created successfully.')        

I used a try statement to make sure that the directory doesn’t exist before creating it. This is because a FileExistsError is raised if the directory already exists. That gives peace of mind.

The os.makedirs method is also used to create directories but it does this recursively. That means, you can use it to create successive directories. The syntax of the method is: os.makedirs(name, mode=0o777, exist_ok=False). Path is the name of the directory you want to create. It has a different argument though from the python make directory method that is worth mentioning. It has an exist_ok keyword argument which you can set to True if you want to create subdirectories of an already existing directory. Let’s use an example:

import os

try:
    os.makedirs('new_dir\\second_dir\\third_dir', exist_ok=True)
except FileExistsError:
    print('Directory already exists.')
else:
    print('Directory created successfully.')        

If you run the above on your machine, it creates the directories second_dir and third_dir (remember new_dir has already been created) and prints: ‘Directory created successfully.’ This is because I set the exist_ok argument to True i.e it should create subdirectories even where a directory already exists. The exist_ok argument comes convenient.

How to remove a directory in python

The methods under this category come in handy when you no longer need a directory. You can programmatically remove directories using python with the python remove directory method, os.rmdir, and the python remove directory recursive method, os.removedirs. The latter removes directories recursively. I didn’t tell you in the earlier post on file handling, but you can also remove files if you want to using the python remove file method, os.remove. I will describe all three here.

To remove a single directory, you use the python remove directory, os.rmdir, method. The syntax of the method is: os.rmdir(path, *, dir_fd=None). Path is the name of the directory you want to remove. With this method, you cannot remove directory trees or directories that are not empty otherwise it will raise OSError exception. If the directory does not exist, it will raise a FileNotFoundError.

When I wanted to remove the new_dir created earlier with child directories like this:

import os

try:
    os.rmdir('new_dir')
except OSError:
    print('Directory not empty.')
else:
    print('Directory successfully removed.')        

It printed out: ‘Directory not empty.’ That means I cannot remove a directory with child directories using this method. Not to worry, the second method, the python remove directory recursive method can do that: os.removedirs

The syntax of the os.removedirs method is: os.removedirs(name) where name is the name of the directory you want to remove.

In the example below, I wanted to remove all the directories and sub-directories we created when making directories.

import os

try:
    os.removedirs('new_dir\\second_dir\\third_dir')
except OSError:
    print('Directory not empty.')
else:
    print('Directory successfully removed.')        

It ran successfully and printed: ‘Directory successfully removed.’ To ensure it doesn’t raise an OSError exception, you should make sure that the leaf directory, third_dir, is empty i.e it doesn’t contain any files.

Now, let’s show the bonus method on how to remove a file.

The method for removing files is the python remove file, os.remove, method. The syntax is: os.remove(path, *, dir_fd=None). Path is the name of the file. If the file is already in use, the method raises an error. Note that the file name, path, should be relative to the current working directory.

In this example here, I want to remove a file that was used when we discussed the file handling methods in an earlier post:

import os

os.remove('eba.txt')
if os.path.isfile('eba.txt'):
    print('File not removed.')
else:
    print('File removed.')    

It ran successfully and printed: ‘File removed.’

How to rename a directory

We can programmatically rename a file or directory in python. There are methods for both single file or directory, or multiple files or directories. The python rename method, os.rename, works for single file or directory, while the python rename recursive, os.renames, method works recursively.

The syntax for the os.rename method is: os.rename(src, dst, *, src_dir_fd=None, dst_dir_fd=None) where src means the source file or directory, and dst means the new name you intend to give the source. The dst or new name should not already exist otherwise the operation will raise an OSError exception or that of one of its subclasses depending on the operating system used.

Here is an example of usage:

import os

try:
    os.mkdir('new_dir')
    print('Directory created successfully.')
    print('Now attempting to rename it.')
    os.rename('new_dir', 'old_lady')
except FileExistsError:
    print('Directory already exists.')
except OSError:
    print('Couldn\'t rename the directory.')
else:
    print('new_dir changed successfully to old_lady.')    

In the example above, I first created a new directory, new_dir, and when it went successfully without raising an error, I then attempted to rename it from new_dir to old_lady. If old_lady already exists, it will raise an OSError exception which I would handle by printing out: ‘Couldn’t rename the directory’ but if it doesn’t exist already, the renaming would run successfully, (which happened) and then print out: ‘new_dir changed successfully to old_lady.’

Now we can do this recursively. What if we create a directory tree with an empty leaf directory. We would have to use the python rename recursive, os.renames, method.

The syntax of the os.renames method is: os.renames(old, new) where old refers to the old name of the directory or directories and new refers to their new names.

Let’s take an example from above again. This time, we want to rename all the directories and sub-directories.

import os

try:
    os.makedirs('new_dir\\second_dir\\third_dir', exist_ok=True)
    print('Directories created successfully.')
    print('Attempting renaming of the directories created.')
    os.renames('new_dir\\second_dir\\third_dir', 'my_first\\my_second\\my_third')
except FileExistsError:
    print('Directories already exists.')
except OSError:
    print('Couldn\'t rename the directories.')
else:
    print('Renamed all three directories recursively.')                

From the above, you could see that I first created a directory tree, new_dir\second_dir\third_dir, and when it was created successfully, I tried an attempt at renaming all the directories recursively using a second try statement. If you do not have the necessary permissions to rename the directory, then the operation will fail. But if the permissions are available and the directories exist as stated in the names for the old directory, then they will be renamed and the code will print: ‘Renamed all three directories recursively.’

You can be creative and try out your own examples to see how it will run.

How to list all the files and nested directories of a directory

I am using windows, so Linux or Unix users pardon me if my example is Windows based. If on windows you want to list the contents of a directory, you use the command ‘dir’ on the command line and it gives you a listing. You can do the same with python. Python has two methods for doing so: a python list directory, os.listdir, method and an optimized python scan directory, os.scandir, method.

It is recommended that you use the python scan directory, scandir, method for most cases, but let me show a working example of the python list directory method. The syntax of the python list directory method is: os.listdir(path='.') where path is the name of the directory whose contents you want to list. The path parameter is optional and where omitted, it defaults to the current working directory. The method returns a list of all the files and directories that are contained in the directory named path.

Here is an example:

import os

dir_list = os.listdir()
for file in dir_list:
    if os.path.isfile(file):
        print(f'{file} is a file.')
    else:
        print(f'{file} is a directory.')    

The above code first returns a list of all the files and directories in the current working directory as dir_list. Then I iterate through the list in a for loop and print out whether an item is a file or a directory. This gives you a listing that is similar to the windows ‘dir’ command line .

Now, for the optimized python scan directory, scandir, method. The syntax of the optimized scan directory method is os.scandir(path='.') where path is the name of the directory. Scandir returns an iterator which yields objects that correspond to the files or nested directories in the path name. You can return the object name, whether they are files or directories, from the objects yielded. (If you want a refresher on iterators or on python generators that yields objects then click on the corresponding links.). Having objects that have file types and attributes increases code performance provided the operating system can provide this information.

Since the iterator produced by the python scan directory method is a resource, you need to close it or garbage collect it by calling the close method, scandir.close(), but you could do this better by using a with statement.

In the example below, we will list the contents of the current working directory again, but this time showing how to do it with the python scan directory method working as a generator.

import os

with os.scandir() as my_dir:
    for item in my_dir:
        if item.is_dir(follow_symlinks=False):
            print(f'{item.name} is a directory.')
        else:
            print(f'{item.name} is a file.')      

I used the with statement so that python will automatically close the iterator immediately the operation ends. You will notice that each object, item, yielded also has attributes of their own. In this example, item object has name attribute in item.name, and also the is_dir method in item.is_dir. This is because the objects are os.DirEntry objects. In the item.is_dir method, in order not to follow symbolic links and list a directory having files as a file, I switched the follow_symlinks parameter to False. This makes it possible to accurately get all directory listings.

Now you have been equipped to use python’s directory handling functions. Go experiment with what you can do with them.

Happy pythoning.