Python Abstraction: A Guide

The notion of abstraction comes from the ability to distil a complicated system into its component parts or fundamental parts. It involves hiding the complexity and showing only the fundamentals. For example, take a car. A car has several functionalities when you enter inside. It has a steering, a brake, a seat, lights etc. When abstracting a car, you break down these component parts and give them names that you use to describe them as well as their functionalities that makes up the whole, the car.

In python, when you specify a data type as an abstract data type, you are specifying the type of data that is stored, the operations supported on it, and the types of the parameters on the operations. This is called the public interface of the abstract data type. And python has a great deal of latitude in providing the specifications for an interface. In python, interface specification uses duck typing. That means, you don’t check the data types at compile time and there are no formal requirements that abstract classes need to be declared. Instead a programmer would have to assume, when in python, that an object supports a known set of behaviors which if they fail the interpreter will raise an error. Duck typing is attributed to a poet, James Whitcomb Riley, who stated that “when I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.”

Mechanism for supporting abstract data types.

Abstraction and interfaces in python are generally implemented by using abstract classes and interfaces. Formally, python provides an abstract base class (ABC). An abstract base class cannot be instantiated, but it defines one or more common methods that all subclasses of the class must have. You realize an ABC by declaring one or more concrete classes that inherit from the abstract base class while providing implementation of the methods declared by the ABC. To declare a class as an abstract base class, ABC, that class needs to inherit from the abc module, a module that provides formal support for ABCs.

When an ABC needs to be an interface, it needs to just provide the method names without any method body. Also, the method is decorated with @abstractmethod to tell python interpreter to look for implementation of that method in the subclasses or concrete classes inheriting from that ABC.

Let’s give an example of data abstraction in python. Let’s say there is a bank providing three methods of payment - payment by credit card, payment by mobile phone, and payment through social media like whatsapp. The banking app could provide a Payment ABC which is declared such and then provides an abstract method, payment, which needs only be implemented by concrete classes inheriting from this payment class.

Let’s code the Payment class ABC as a python abstract class.

    
from abc import ABC, abstractmethod

class Payment(ABC):

    def print_receipt(self):
        print('Receipt printed for payment', 
                  self.__class__.__name__)

    @abstractmethod
    def payment(self):
        pass 

You can see from the code above that I first imported ABC from the abc module. Then the Payment class inherits ABC class in that module, making it a python abstract class. The Payment class has two methods: a concrete method, print_receipt, that applies to all instances of the class including subclasses and an abstract method, payment, which will have to be implemented by concrete classes inheriting from it. I decorated the abstract method with the @abstractmethod decorator to tell python what to expect. Notice that in the print_receipt method, I made a reference to the class name of the object calling the method so that we can be able to find out which subclass we are printing the receipt for.

Now, let’s write concrete classes that inherit from the Payment class and see how they implement the payment method, abstracting its functionality so they can define their own way of payment.

You can see now that each subclass only needs to implement the method declared as an abstract method. If they don’t, python will give an error. After implementing that class, they behave differently while still being recognized as members of the Payment class.

Whenever an object which implements the abstract method invokes the abstract method, its own implementation is called and executed. That’s how python abstraction works. So, you don’t have to be peering about what is happening under the hood because all you need to know is that mobile payment, credit card payment and whatsapp payment work.

I believe you now have a good working knowledge of how abstraction in python works and can implement your own abstract classes and interfaces. Data abstraction in python allows your class to have several objects doing the same thing in their own distinct way. You need not know how it works but just that it works. Isn’t data abstraction in python beautiful? It’s a cool feature of the language.

In fact, python abstraction is one of the OOP concept every programmer needs to know. I have covered other OOP concepts like inheritance and also polymorphism in other posts.

Happy pythoning.

Constructing An XML Parser In Python

XML, or extensible markup language, is a markup language defining a set of rules for encoding documents such that they can be both human-readable and machine-readable. The World Wide Web Consortium has published a set of standards that define XML. You can reference the specifications here. Although XML was initially designed for documents, its use case has included several types of media and files.

 

A well-formed XML document among other things would include elements and attributes. An element, like in HTML, is a logical document component that begins with a start-tag and ends with an end-tag. A start-tag is denoted as <tag_name> while the end-tag as </tag_name>. An empty tag is a combination of both and is denoted as <tag_name />. An element could also have attributes within the start-tags or empty tags. Attributes are name-value pairs for the document and each name can only have one value. Example of an element with an attribute is <subtitle lang=’en’> where the subtitle element has the lang attribute with ‘en’ value. At the top of the XML document is a root element which is the entry into the document.

Now, in our code that parses XML we will only be dealing with elements and attributes.

To parse XML, python has an API for doing that. The module that implements the API is the xml.etree.ElementTree module, so you would have to import this module into your python file to use the API.

What the xml.etree.ElementTree module contains

This module is a simple API for parsing and creating XML in python. Although it is robust, it is not secure against maliciously constructed data. So, take note. Among several classes of interest, for our parsing activity we will be concentrating on two classes in this module – ElementTree which represents the whole XML document as a tree, and Element which represents a single node in the tree.

To import an XML document you could import it from a file or pass it as a string.

To import it from a file use the following code:

    
import xml.etree.ElementTree as etree
tree = etree.parse('data.xml')
root = tree.getroot()

while to get it directly from a variable as a string use the following code:

    
import xml.etree.ElementTree as etree
xml = 'data as string'
root = etree.fromstring(xml)

The root variable above refers to the root element in the XML document.

The ElementTree constructor

We will be using the ElementTree constructor to get to the root of our XML document, so it is worth mentioning here. The syntax for the constructor is xml.etree.ElementTree.ElementTree(element=None, file=None). The constructor can accept an element which serves as the root element as argument or you could pass it a file that contains the XML document. What it returns is the XML document as a tree that could be interacted with.

One interesting method of this class is the getroot() method. When you call this method on an ElementTree root, it returns the root element in the XML document. We will use the root element as our doorway into the XML document. So, take note of this method because we will be using it in our parsing code below.

That’s all we need from ElementTree class. The next class we will need is the Element class.

Objects of the Element class.

This class defines the Element interface. It’s constructor is xml.etree.ElementTree.Element(tag, attrib={}, **extra). But we will not be creating any elements but just using the attributes and methods. Use the constructor to create an element. But you can see from the constructor definition that an XML document element has two things: a tag and a dictionary of attributes. Objects of this class defines every element in the XML document.

Some interesting attributes and methods we will be using from this class are:

a. Element.attrib: This returns a dictionary that represents the attributes of the said element. What is included in the dictionary are name-value pairs of attributes in the Element or what some call Node in an XML document.

b. Element.iter(tag=None): this is the iterator for each element. It recursively iterates through the children of the element and gives you all the children, even the children of its children recursively. You could filter which result it can give by providing a tag argument that specifies the specific tag whose children you want to receive information about. It iterates over the element’s children in a depth-first order. But if you do not want to get the children in a recursive fashion but only want the first level children of any element, then you can use the next method below.

c. List(element): This is casting an element to a list. This casting returns a list of all the children, first level only, of the element. This method replaces the former Element.getchildren() method which is now deprecated.

So, I believe you now have a simple introduction into some of the features of the xml.etree.ElementTree module. Now, let’s implement this knowledge by parsing some XML documents.

The XML document we are going to parse is a feed for a blog. The XML document is given below:

    
<feed xml:lang='en'>
        <title>SolvingIt?</title>
        <subtitle lang='en'>
               Programming and Technology Solutions
                     </subtitle>
        <link rel='alternate' type='text/html' 
         href='https://emekadavid-solvingit.blogspot.com' />
        <updated>2020-09-12T12:00:00</updated>
        <entry>
            <author>
                <name>Michael Odogwu</name>
                <uri>
                https://emekadavid-solvingit.blogspot.com
                </uri>
            </author>
        </entry>
    </feed>   

You can reference this document in the code while reading the code. You can see that the XML document has elements or nodes and the root tag is named feed. The elements also have attributes.

The first task we are going to do is that we are going to find the score of the XML document. The score of the XML document is the sum of the score of each element. For any element, the score is equal to the number of attributes that it has.

The second task is to find the maximum depth of the XML document. That is, given an XML document, we need to find the maximum level of nesting in it.

So, here is the code that prints out the score and maximum depth of the XML document above. I want you to run the code and compare the result with what you would have calculated yourself. Then, after running the code, the next section is an explanation of relevant points in the code along with a link to download the script if you want to take an in-depth look at it.

Now, for an explanation of the relevant sections of the code. I will use the lines in the code above to explain it.

Line 1: We import the module, xml.etree.ElementTree and name it etree.

Lines 23-35: The XML document.

Line 36, 37: Using the fromstring method of the module, we import the xml document and pass it to the ElementTree constructor which then constructs a tree of the document. Then from the tree created we get the root element (or node) so that we can parse the document starting from the root element.

Line 39: We pass the root element to our function, get_attr_number, that calculates the score of the XML document.

Lines 3-8: What the get_attr_number function does is that it takes the root element or node and recursively iterates through it using node.iter() to get all the children, even the nested children. For each child element, it calculates the score for that child by finding out the length of the attribute dictionary in it, len(i.attrib) and then adds this score to the total score. It then returns the total score as the total variable.

Next is to find the maximum depth. In the XML tree, we take the root element, feed, to be a depth of 0. Take note.

Lines 41,42: Here the depth function is called, passing it the root element of the tree and the default level is noted as -1. Then maxdepth, a global variable, is printed out after the depth function has finished execution. I now describe the depth function.

Lines 12-20: When this function is called, it increases the level count by 1 and checks to see if the level is greater than the maxdepth variable in order to update maxdepth. Then for each node or element, if that element has children, list(elem), it calls the function, depth, recursively.

You can download the above code here, xmlparser.py.

Now, I believe you understand how the code works. I want you to be creative. Think of use cases of how you can use this module with other XML functions like creating XML documents, or writing out your own XML documents and parsing them in the manner done above. You can also check out another parser I wrote, this time an HTML parser.

Happy pythoning.

Validating Credit Cards With Python Regex

We have been exploring python regex in previous posts because it is an interesting area of programming. It gives me the adrenaline surge when I get a pattern right. I usually use regex101.com to confirm my patterns in python regex.

 

Because I have touched extensively on the syntax of python regex in previous posts, like on the syntax of python regex and the methods that are used in python regex, I will go straight to describing today’s task.

In today’s coding challenge, we want to validate credit card numbers. We know credit cards are 16 digits long, but what goes into their creation? For a credit card number to be valid, it needs to have the following characteristics:

1. It must start with 4, 5, or 6 (assuming that is the requirement for our bank MZ. Other banks have different requirements but we can scale)

2. It must contain exactly 16 digits.

3. It must only consist of digits, i.e, 0 – 9.

4. It may have digits in groups of 4 separated by a single hyphen, ‘-‘.

5. It must not use any other separator except the hyphen.

6. It must not have 4 or more consecutive repeated digits.

Now, that list of requirements is long. Yes, it is long. But we can scale to the requirements. Here is code that does just that. I would like you to read the code and then run it to see the output. Later, I will explain each of the regex patterns and what the code is doing.

Now that you have read it and run it, I sure hope you understand the code. Not to worry, I am here to walk you through the lines of the code. But first, let me explain some relevant details in regex that will help you understand the code. That is, the python regex meta characters that would be of help.

?	This is called the optional match. This tells python to match 0 or 1 repetitions of the preceding regular expression.
{m}	This tells python to match exactly m repetitions of the preceding regular expression.
[abc]	Used to indicate a set of characters. Any character within the set is matched. This example matches either a, b, or c that are within the set.
(…)	This matches whatever regular expression is inside the parenthesis and indicates the start and end of a group. The contents of the group can be retrieved after the match has been performed and can be matched later in the string with the \number special sequence.
\number	Matches the contents of the group with this number. Group numbering starts from 1.
(?!...)	matches if … doesn’t match next. This is the negative lookahead assertion. For example Isaac(?!Asimov) will match Isaac only if it is not followed by Asimov.

Now that I have explained all the relevant meta characters you need to understand the code, let’s go through the code, starting first with the patterns for the match.

On lines 4 and 5, you can see that I wrote two patterns we will be using to do the matches.

Line 4, the valid_structure pattern: r"[456]\d{3}(-?\d{4}){3}$”. First it indicates a set of characters to start off the pattern, [456]. That means the start of the pattern should be a 4, 5, or 6 based on our credit card characteristic. Then this should be followed by exactly 3 digits. \d indicates digits. After this we have a group which consists of an optional hyphen, -, and then exactly 4 digits. This group should be matched three times. When we calculate the digits together, this goes to 16. So, with the valid_structure pattern, we have satisfied nearly all the requirements, except the requirement that there should be no 4 consecutive repeats of any digit.

That is where no_four_repeats pattern comes in, on line 5. The pattern is r"((\d)-?(?!(-?\2){3})){16}". Now let’s go through it. First we did a global grouping. This is group 1. Then we grouped the digit at the start of the pattern; it will become group 2. What the pattern is saying is that a digit could be followed by a hyphen. Then we did a negative lookahead assertion in a group. In the negative lookahead assertion we said that the group should not include an additional hyphen and other digits exactly three times. What we are saying is that the grouping in the negative lookahead assertion should not exist in the string and if it does, there is no match. Then all the digits in the group should be exactly 16 digits. If you want a refresher on negative lookahead assertions, you can check this blog post on assertions in regular expressions.

The rest of the function following from the patterns is self explanatory. We pack the patterns into a tuple in line 6. From lines 8 to 12 we search for a match based on the patterns and the list of credit cards that was passed.

I hope you did find the code interesting. It was beautiful matching credit card numbers. I hope to bring you more like this if I find any challenge that seems interesting.

If you would like to receive instant updates when I post new blog articles, subscribe to my blog by email.

Happy pythoning.

First Tracking Device using Vibration, AI to track 17 Home appliances

At the present state of affairs, to track each appliance in your home you would need to install a separate tracker for each appliance. Now, what if you had 10, 20 or so appliances? That would be some expense to carry out, not so? But recently, some researchers at Cornell University have developed a single device that is able to track about 17 home appliances at the same time and this device uses vibration with an integrated deep learning network. With this device, you no longer need to worry about forgetting to take wet clothes out of the washing machine, or allowing food to remain in the microwave, or even forgetting to turn off faucets that are dripping. This device promises to make your home smart in a cost-effective way.

 

Vibration analysis has several uses in industry, especially in detecting anomalies in machinery, but this is the first use case for tracking home appliances using vibrations that I have found. This device, called Vibrosense, uses lasers to capture the subtle vibrations that are emitted by walls, floors and ceilings and then incorporates this received vibration with a deep learning network that is used to model the data being processed by the vibrometer in order to create a unique signature for each appliance. I tell you, researchers are getting closer to their dream of making our homes not only smarter, but more efficient and integrated.

But can it detect appliance usage across a house, you may ask? There are so many appliances in a house and the vibrations they emit can intersect. That’s right. The researchers have a solution to that problem. To efficiently detect different appliances in a house and not just in any single room, the researchers divided the task of the tracking device into two categories: First, the tracking device would have to detect all the vibrations in the house generally using the laser Doppler vibrometer, and second differentiate the vibrations from multiple appliances even if they were similar vibrations by identifying the path the vibrations has traveled from room to room.

The deep learning network that is incorporated in the device uses two modes of learning: path signatures and noises. Path signatures for identifying different activities and the distinctive noises that the vibrations make as they travel through the house.

To test its accuracy the tracking device was tested across 5 houses at the same time and it was able to identify the vibrations from 17 different appliances with 96% accuracy. Some of the appliances it could identify were dripping faucets, an exhaust fan, an electric kettle, a refrigerator, and a range hood. Also, when it was trained Vibrosense could be able to identify 5 stages of appliance usage using an accuracy of 97%.

Cheng Zhang, assistant professor of information science at Cornell University and director of Cornell’s SciFi Lab, on speaking about the device, Vibrosense, said that it was recommended for use in single-family houses because when it was installed in buildings, it could pick up the activities that were going on in neighboring houses. A big privacy risk one must say.

A smart device with immense benefits

When computers are able to recognize the activities going on in the home, it makes our dream of the smart home closer to reality. Such computers can ease the interaction between humans and computers, enabling human-computer interfaces that are a win-win for everyone. That is what this tracking device does. One advantage of this device is that it leverages on the use of computers to understand human needs and behaviors. Formerly, we would need separate devices for each appliance or need. But this device has leveraged on that need. “Our system is the first that can monitor devices across different floors, in different rooms, using one single device,” Zhang said.

I feel elated on discovering this device. No more having to wait for my food to be cooked on the microwave. With this device, I could be watching the TV while it watches the food on my behalf. There are a lot of things we could use this for. I think this innovation is very beneficial to the average American.

But one concern about Vibrosense is in the area of privacy. I wouldn’t want my neighbor to know when I am in the bathroom, or have the TV on, or that I was not in the house. But these are the information the device can send out.

When asked on the issue of privacy, Zhang said: “It would definitely require collaboration between researchers, industry practitioners and government to make sure this was used for the right purposes.” I hope that cooperation does come.

The device could even help in enabling sustainability and energy conservation in the home. In so doing, it could help homes to monitor their energy usage and reduce consumption. It could also be used to estimate electricity and water usage rates since the device has the ability to detect both the occurrence of an event and the exact time period that event took place. This is badly sought-for energy-saving advice that home owners need. This is great!

I was thinking about the benefits of a device like this in a typical home and was wowed by its potential benefit that I decided that this innovation needs a place in my solvingit? blog. So, this is a thumbs up to Cheng Zhang and his team at Cornell.

The material for this post was based on the paper: “VibroSense: Recognizing Home Activities by Deep Learning Subtle Vibrations on an Interior Surface of a House from a Single Point Using Laser Doppler Vibrometry.” Cheng Zhang was senior author of the paper. The paper was published in Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies and will be presented at the ACM International Joint Conference on Pervasive and Ubiquitous Computing, which will be held virtually Sept. 12-17.

Techniques for packing and unpacking in python

When I first started learning python, one of the cool things I learned was packing and unpacking of sequences like tuples. I fell in love with them that I ended up using them all the time. They were a feature I had found only in python and javascript.

 

Packing and unpacking, also called destructuring i.e the ability to extract multiple values from data stored in objects, has lots of techniques I have discovered over the years. I will highlight some in this post.

a. Packing and unpacking of sequences

As a convenient way to assign variables with a single line of code, automatic packing is an often used technique that can serve you well. It usually is applied to sequences like tuples, lists, and ranges. For a refresher on sequences and how they are a subset of iterables, see this post.

For example, we could automatically pack and unpack this list into three variables, x, y, and z.

We could do a similar thing with a tuple, unpacking the tuple and assigning its values.

Most often where programmers apply this automatic packing and unpacking technique is when they want to return multiple values from a function. Take this function for quotients and remainder that returns a tuple.

You can see that when the call returns the output, a tuple, is automatically packed into two variables. You get to see this often in code and this is a cool feature from python.

The packing feature has its dual companion, the unpacking feature for sequences. I highlighted one above but let me show you another with unpacking in tuples.

See how python nicely unpacks the range and assigns it immediately to the variables on the left.

One thing you need to note when packing and unpacking with sequences is that the number of variables on the left should match the number of items you want to unpack or pack on the right.

You do not only pack and unpack on the right side of the assignment operator. You can also pack and unpack on the left. Take this example.

I first assigned the first element to variable ‘a’ and then packed the remaining items in the tuple into ‘d’. Just use your creativity.

You can unpack in a for loop. Remember, for loops need an iterable. That is why unpacking can be deployed there also. Take this list of tuples and see how one can unpack the tuples in a for loop.

This technique of unpacking in a for loop is what you are doing when you call the dictionary.items() method with a syntax like this:

This technique of packing and unpacking has resulted in the tradition of simultaneous assignments in python which has come to replace the need for temporary variables when you want to swap values among two variables. For example, in the former archived way of swapping, one could do it this way:

    
temp = y
y = x 
x = temp

This has come to be replaced with the more convenient and readable simultaneous assignment syntax that goes like this:

x, y = y, x

Don’t you just love the way python is evolving? I just adore it.

Now, packing and unpacking is not restricted to sequences. It extends to the arena of function arguments and function calls.

2. Packing and unpacking in functions

There are several techniques for packing and unpacking in functions and I will highlight some of the common ones.

a. Python argument tuple packing and unpacking

To carry out a packing of the arguments that are passed to the function, you just precede the formal parameter with an asterisk, *. Then any value or object that is passed to the function will be packed into a tuple.

Here is an example.

Because this argument packing can accept multiple types, you would be wise to assert the types you desire in your function.

Python argument tuple packing has its counterpart, python argument tuple unpacking. This is when on the calling side of the function you precede the object with an asterisk. The object would usually be a sequence or iterable. Then when the object is passed to the function it is unpacked.

Take this example:

Note that the number of formal arguments should be equal to the object when it is unpacked.

But who says we cannot do unpacking and packing at the same time. Yes, you can do packing and unpacking in function calls at the same time.

Let’s go to another data structure that is not a sequence – a dictionary.

b. Python argument dictionary packing and unpacking.

Just as you can do packing and unpacking for sequences, you can also do the same for dictionaries, but in the case of dictionaries you use double asterisks, **, to indicate that you are dealing with a key=value pair. While reading the python documentation several times, even on pylint if your IDE has pylint installed, you would find formal arguments specified as **kwargs. The **kwargs formal argument just says that the object that is passed would be packed into a dictionary.

Let’s take an example of a common case.

The keywords passed in the function call can be any number but the function will pack all of them into a dictionary that fits.

Now let’s illustrate the argument dictionary unpacking. This occurs at the opposite end, at the function call. This is when the object passed is preceded by double asterisk. This tells the function that the object, which is a dictionary, should be unpacked.

One final example.

f(**names) is just a shorthand for the explicit keyword arguments f(x=’A’, y=’A+’, z=’C’).

There are so many techniques which have sprung up from the principle of packing and unpacking iterables. If you have some you want to share which I didn’t mention, make a comment below and I will be sure to update this post.

Thanks. Happy pythoning.