#!/usr/bin/env python
# coding: utf-8

# ---   
#  <img align="left" width="75" height="75"  src="https://upload.wikimedia.org/wikipedia/en/c/c8/University_of_the_Punjab_logo.png"> 
# 
# <h1 align="center">Department of Data Science</h1>
# <h1 align="center">Course: Tools and Techniques for Data Science</h1>
# 
# ---
# <h3><div align="right">Instructor: Muhammad Arif Butt, Ph.D.</div></h3>    

# <h1 align="center">Lecture 2.18</h1>

# <a href="https://colab.research.google.com/github/arifpucit/data-science/blob/master/Section-2-Basics-of-Python-Programming/Lec-2.18-Regular-Expressions-in-Python/Regular-Expressions-Python.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# ## _Regular Expressions Part-II.ipynb_

# <img align="left" width="600" height="600"  src="images/re.jpeg"  >
# <img align="right" width="300" height="300"  src="images/tm.jpg"  >

# ## Learning Agenda
# 1. Review of Meta Characters, Quantifiers and Escape Codes
# 2. The Python `re` Module
#     1. The `re.compile()` Method
#     2. The `re.Pattern.search()` Method
#     3. The `re.Pattern.match()` Method
#     4. The `re.Pattern.findall()` Method
#     5. The `re.Pattern.finditer()` Method
# 3. Practical Example
#     1. Extracting Names
#     2. Extracting Date of Births
#     3. Extracting Emails and Usernames
#     4. Extracting valid Cell phones
#     5. Extracting Domain names from URLs
# 4. Modifying Strings
#     1. The `re.Pattern.split()` Method
#     2. The `re.Pattern.sub()`  Method
#     3. The `re.Pattern.subn()` Method

# ## 1. Review of Meta Characters, Quantifiers and Escape Codes

# ### a. Wild Card / Meta Characters
# Special characters are characters that do not match themselves as seen but have a special meaning when used in a regular expression. Some commonly used wild cards or meta characters are listed below:
# 
# 
# | Wild Card | Description         
# | :-:       |:-------------
# | **^**     |Caret symbol specifies that the match must start at the beginning of the string, and in MULTILINE mode also matches immediately after each newline<br>- `^b` will check if the string starts with 'b' such as baba, boss, basic, b, by, etc.<br>- `^si` will check if the string starts with 'si' such as simple, sister, si, etc.
# | **$**     |Specifies that the match must occur at the end of the string <br> - `s$` will check for the string that ends with a such as geeks, ends, s, etc.<br>- `ing$` will check for the string that ends with ing such as going, seeing, ing, etc.
# | **.**     |Represent a single occurrance of any character except new line <br> - `a.b` will check for the string that contains any character at the place of the dot such as acb, acbd, abbb, etc<br> - `..` will check if the string contains at least 2 characters
# | **\\**    |Used to drop special meaning of a character following it or used to refer to a special character. <br> - Since dot `(.)` is a metacharacter, so if you want to search it in a string you have to use the backslash `(\)` just before the dot `(.)`  so that it will lose its specialty. 
# | **[...]** |Matches a single character in the listed set. If caret is the first character inside it, it means negation<br>- `[abc]` means match any single character out of this set<br>- `[123]` means match any single digit out of this set<br>- `[a-z]` means match any single character out of lower case alphabets<br>- `[0-9]` means match any single digit out of this set<br>- `[^0-3]` means any number except 0, 1, 2, or 3<br>- `[^a-c]` means any character except a, b, or c<br>- [0-5][0-9] will match all the two-digits numbers from 00 to 59<br>- `[0-9A-Fa-f]` will match any hexadecimal digit.<br>- Special characters lose their special meaning inside sets, so `[(+*)]` will match any of the literal characters '(', '+', '*', or ')'.<br>- To match a literal ']' inside a set, precede it with a backslash, or place it at the beginning of the set, so `[()[\]{}]` and `[]()[{}]` will both match parenthesis.
# | **^[...]**|Matches any character in the set at the beginning of the string
# | **[^...]**|Matches any character except those NOT in the listed set (negation)
# | **\|**    |Or symbol works as the OR operator meaning it checks whether the pattern before or after the or symbol is present in the string or not<br>- `a\|b` will match any string that contains a or b such as acd, bcd, abcd, etc.<br>- To match a literal '\|', use `\|`, or enclose it inside a character class, as in `[\|]`.
# | **( )**   |Used to capture and group

# ### b. Quantifiers
# - A quantifier metacharacter immediately follows a portion of a <regex> and indicates how many times that portion must occur for the match to succeed. *, +, ?, {m}, {m,n}. When used alone, the quantifier metacharacters *, +, and ? are all greedy, meaning they produce the longest possible match. 
# 
# | Wild Card | Description         
# | :-:       |:-------------
# | **\***    |The preceding character/expression is repeated zero or more times
# | **+**     |The preceding character/expression is repeated one or more times, <br>- `ab+c` will be matched for the string abc, abbc, dabc, but will not be matched for ac, abdc because there is no b in ac and d is not followed by c in abdc.
# | **?**     |The preceding character/expression is optional (zero or one occurrence). <br>- `ab?c` will be matched for the string ac, abc, acb, dabc, dac but will not be matched for abbc because there are two b. Similarly, it will not be matched for abdc because b is not followed by c.
# | **{n,m}** |The preceding character/expression is repeated from n to m times (both enclusive). <br> - `a{2,4}` will be matched for the string aaab, baaaac, gaad, but will not be matched for strings like abc, bc because there is only one a or no a in both the cases.
# | **{n}**   |The preceding character/expression is repeated n times.<br>- `a{6}` will match exactly six 'a' characters, but not five.           
# | **{n,}**  |The preceding character/expression is repeated atleast n times 
# | **{,m}**  |The preceding character/expression is repeated upto m times
#     
#     
# Note: The repeat characters (`*` and `+`) perform greedy search to match the largest possible string. However, you can performa a non greedy search by: `*?` (0 or more characters but non-greedy) `+?` (1 or more characters but non-greedy)

# ### c. Escape Codes
# - You can use special escape codes to find specific types of patterns in your data, such as digits, non-digits,whitespace, and more. 
# - The following list of special sequences isn’t complete.
# 
# | Code | Description         
# | :-:  |:-------------
# | **\d** |Matches any decimal digit. This is equivalent to [0-9]                              
# | **\D** |Matches any non-digit character. This is equivalent to [^0-9] or [^\d]                           
# | **\s** |Matches any whitespace character. This is equivalent to [ \r\n\t\b\f]                
# | **\S** |Matches any non-whitespace character. This is equivalent to [^ \r\t\n\f] or [^\s]                         
# | **\w** |Matches alphanumeric character. This is equivalent to [a-zA-Z0-9_]                  
# | **\W** |Matches any non-alphanumeric character. This is equivalent to [^a-zA-Z0-9_] or [^\w]                  
# | **\b** |Matches where the specified characters are at the beginning or at the end of a word r"\bain" OR r"ain\b"
# | **\B** |Matches where the specified characters are present, but NOT or at the end of a word r"Bain" OR r"ain\B" 

# ## 2. The Python `re` Module

# In[11]:


import re
print(dir(re))


# ### a. The `re.compile()` Method
# This method is used to compile a regular expression pattern into a regular expression object, which can be used for matching using its `match()`, `search()` and other methods.
# 
# **`re.compile(pattern, flags=0)`**
# 
# Where,
#    - `pattern` is the regular expression which you want to compile that you need to search/modify in a string or may be on a corpus of documents.
#    - `flags` arguments can be used to modify the expression’s behaviour. Values can be any of the following variables, combined using bitwise OR (the | operator):
#         - `IGNORECASE` or `I` to do a case in-sensitive search
#         - `LOCALE` or `L` to perform a locale aware match.
#         - `MULTILINE`, `M` to do multiline matching, affecting `^` and `$`
#         - `DOTALL` or `S` to make the '.' special character match any character, including a newline; without this flag, '.' will match anything except a newline.
# 
# Once you have an `Pattern object` representing a compiled regular expression, you can use its methods to perform various operations on a string or may be in a corpus of documents:
# - `p.match()`: Determine if the RE matches at the beginning of the string.
# - `p.search()`: Scan through a string, looking for any location where this RE matches.
# - `p.findall()`: Find all substrings where the RE matches, and returns them as a list.
# - `p.finditer()`: Find all substrings where the RE matches, and returns them as an iterator.
# - `p.split()`: Used to split string by the occurrences of pattern.
# - `p.sub()`: Used for for find and replace purpose..
# 
# **Note:** Creating a regex object with `re.compile` is highly recommended if you intend to apply the same expression to many strings; doing so will save CPU cycles.

# In[12]:


import re
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by 
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)  

print(p)
print(type(p))


# Once you have got a pattern object, you can use its various methods for searching from a string

# In[ ]:





# ### b. The `re.Pattern.search()` Method
# - Scan through string looking for the first location where the pattern object `p` produces a match, and return a corresponding match object. Return None if no position in the string matches the pattern.
# 
# **`p.search(string, pos=0 endpos=9223372036854775807)`**
# 
# - Where,
#    - `p` is the compiled pattern object
#    - `string` is the test string from which we want to search
#    - `pos` and `endpos` can be used to specify the portion of test string from where to search

# In[13]:


import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by 
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)  

match = p.search(str1)
print(match)
print(type(match))


# In[ ]:





# ### c. The `re.Pattern.match()` Method
# - Look for the pattern at the beginning of the string and if found returns a corresponding match object. Return None if the string does not match the pattern.
# 
# **`p.match(string, pos=0 endpos=9223372036854775807)`**
# 
# - Where,
#    - `p` is the compiled pattern object
#    - `string` is the test string from which we want to search
#    - `pos` and `endpos` can be used to specify the portion of test string from where to search
# 
# Note: 
# - Even in MULTILINE mode, `re.match()` will only match at the beginning of the string and not at the beginning of each line.
# - If you want to locate a match anywhere in string, use `search()` instead 

# In[15]:


import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by 
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)  

rv = p.match(str1)
print(rv)
print(type(rv))


# In[ ]:





# ### d. The `re.Pattern.findall()` Method
# - The `search()` only returns the first match, `match()` only matches at the beginning of the string, while `findall()`  returns all matches in a string.
# - Return all non-overlapping matches of pattern in string, as a list of strings or tuples. The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result.
# - If pattern `p` does not match, it returns an empty list.
# 
# **`p.findall(string, pos=0 endpos=9223372036854775807)`**
# 
# - Where,
#    - `p` is the compiled pattern object
#    - `string` is the test string from which we want to search
#    - `pos` and `endpos` can be used to specify the portion of test string from where to search
#    
# Note: The result depends on the number of capturing groups in the pattern. If there are no groups, return a list of strings matching the whole pattern. If there is exactly one group, return a list of strings matching that group. If multiple groups are present, return a list of tuples of strings matching the groups. Non-capturing groups do not affect the form of the result.

# In[16]:


import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by 
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)  


rv = p.findall(str1)
print(rv)
print(type(rv))


# In[ ]:





# ### e. The `re.Pattern.finditer()` Method
# - Return an iterator yielding match objects over all non-overlapping matches for the RE pattern in string. The string is scanned left-to-right, and matches are returned in the order found. Empty matches are included in the result.
# 
# **`p.finditer(string, pos=0 endpos=9223372036854775807)`**
# 
# - Where,
#    - `p` is the compiled pattern object
#    - `string` is the test string from which we want to search
#    - `pos` and `endpos` can be used to specify the portion of test string from where to search
#    

# In[17]:


import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."

# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by 
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)  

matches = p.finditer(str1)
print(matches)
print(type(matches))


# >- **Once we have got the iterator of `Match object`, we can iterate it using a `for` loop.**
# >- **Let us see how many match objects are there in this iterator named `matches`.**

# In[18]:


for m in matches:
    print(m)


# In[ ]:





# >- **Every match object has many associated methods.**
# >- **Let us see different attributes of each match object using these methods.**

# The **`group()`** method of the match object, return subgroups of the match (if they exist). By default return the entire match.

# In[22]:


import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."

p = re.compile(r"[A]+[a-z]*")  
matches = p.finditer(str1)

for m in matches:
    print(m.group(0))


# The **`span()`** method of the match object, return a 2-tuple containing the start and end index (end index not inclusive)

# In[23]:


import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."

p = re.compile(r"[A]+[a-z]*")  
matches = p.finditer(str1)

for m in matches:
    print(m.span())


# The **`start(group=0)`** method of the match object, return index of the start of the substring matched by group.

# In[24]:


import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]+")  
matches = p.finditer(str2)

for m in matches:
    print(m.start())


# The `end(group=0)` method of the match object, return index of the end of the substring matched by group.

# In[25]:


import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games.  AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]+")  
matches = p.finditer(str2)

for m in matches:
    print(m.end())


# In[ ]:





# ## 3. Practical Example

# **Read a text file**

# In[27]:


get_ipython().system(' cat datasets/names_addresses.txt')


# In[26]:


with open("datasets/names_addresses.txt", "r") as fd:
    print(fd.read())


# In[ ]:





# **Let us read the data from the file in a string**

# In[28]:


with open("datasets/names_addresses.txt", "r") as fd:
    teststring = fd.read()
teststring


# In[ ]:





# ### a. Extracting Names
# - Assume that every name starts with Mr or Ms or Mrs, with an optional dot, a space and then followed by alphanumeric characters

# In[ ]:


import re
p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+')  

#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
    print(match)


# In[ ]:


p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+')  

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())
    


# What if we want to get the complete name

# In[31]:


p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+\s[A-Za-z]*')  

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())
    


# In[ ]:





# ### b. Extracting Date of Births
# - Assume that the the date, month and year are of two, two and four digits respectively. Moreover, they are separated by either a dot, a hyphen or a slash

# In[ ]:


p = re.compile(r'\d{2}.\d{2}.\d{4}')  

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())
    


# What if we just want to get the date, month and year separately

# In[37]:


p = re.compile(r'(\d{2}).(\d{2}).(\d{4})')  

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())
    


# In[ ]:





# ### c. Extracting Emails and Usernames
# **Valid Name Part:**
# - Lowercase case alphabets
# - Uppercase case alphabets
# - Digits: 0123456789,
# - dot: . (not first or last character)
# - For simplicity assume no special characters allowed
# 
# **Valid Domain Part:**
# - Lowercase case alphabets
# - Uppercase case alphabets
# - Digits: 0123456789,
# - Hyphen: - (not first or last character),
# - Can contain IP address surrounded by square brackets: test@[192.168.2.4] or test@[IPv6:2018:db8::1].

# In[ ]:


get_ipython().system('cat datasets/names_addresses.txt')


# In[ ]:


pattern = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-z]{2,3}')

#teststring contains data read from file
matches = pattern.finditer(teststring)

for match in matches:
    print(match.group())


# What if we want to just extract the usernames

# In[40]:


p = re.compile(r'([a-zA-Z0-9_.+-]+)@([a-zA-Z0-9-]+)(\.[a-z]{2,3})')

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())


# In[ ]:





# ### d. Extracting valid Cell phones
# Assume that every valid phone number consiste of 10 digits in three groups of three, three and four digits. The three groups are separated by either a `-`, `.` or `/` symbol

# In[ ]:


p = re.compile(r'\d{3}[./-]\d{3}[./-]\d{4}')

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())
    


# You can easily extract the city codes, country codes and so on at your own by creating groups inside your regular expressions.

# In[ ]:





# ### e. Extracting Domain names from URLs
# - Assume simple URLs, having the protocol either `http://` or `https://`
# - Then we have optional `www.` string
# - Then we have group of characters that make up our domain name, followed by top level domain

# In[ ]:


p = re.compile(r'https?://(www\.)?(\w+)(\.\w+)')

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group())   


# In[ ]:





# Let us extract the top level domain (TLDs) only.

# In[ ]:


p = re.compile(r'https?://(www\.)?(\w+)(\.\w+)')

#teststring contains data read from file
matches = p.finditer(teststring)

for match in matches:
    print(match.group(3))   


# In[ ]:





# ## 4. Modifying Strings
# - Up to this point, we’ve simply performed searches against a static string. Regular expressions are also commonly used to modify strings in various ways, using the following pattern methods:
# 
#         - split(): Split the string into a list, splitting it wherever the RE matches
#         - sub(): Find all substrings where the RE matches, and replace them with a different string
#         - subn(): Does the same thing as sub(), but returns the new string and the number of replacements

# ### a. The `re.Pattern.split()` Method
# - It split the target string as per the regular expression pattern, and the matches are returned in the form of a list.
# **`p.split(string, maxsplit=0)`**
# 
# - Where,
#     - `string`: The variable pointing to the target string (i.e., the string we want to split).
#     - `maxsplit`: The number of splits you wanted to perform. If maxsplit is 2, at most two splits occur, and the remainder of the string is returned as the final element of the list.
#     
# 
# Note: 
# It’s similar to the `split()` method of strings but provides much more generality in the delimiters that you can split by; string split() only supports splitting by whitespace or by a fixed string.

# In[43]:


# defining string
mystring = "My name             is Arif Butt and my lucky number is 54 and 143"
mylist = mystring.split(sep=' ')
mylist


# In[45]:


# importing required libraries
import re

# defining string
mystring = "My name         is   Arif Butt and my lucky number is 54 and 143"

p = re.compile(r"\s+")

word_list = p.split(mystring)

print(word_list)


# In[ ]:





# The `maxsplit` parameter of `split()` is used to define how many splits you want to perform. In simple words, if the maxsplit is 2, then two splits will be done, and the remainder of the string is returned as the final element of the list.

# In[48]:


import re

# defining string
mystring = "My name     is  Arif Butt and my lucky number is 54 and 143"

p = re.compile(r"\s+")

word_list = p.split(mystring, maxsplit=0)

print(word_list)


# - The `split()` method of strings allows you to split by whitespace or by a fixed string.
# - The regex `split()` method allows you to specify a regex pattern for the delimiters where you can specify multiple delimiters.
# - For example, using the regular expression re.split() method, we can split the string either by the `comma` or by `space`.

# - Let us split by the `comma` or by `hyphen`.

# In[49]:


import re

# defining string
mystring = "12,45,78,85-17-89"

p = re.compile(r"-|,")
p = re.compile(r"[-,]")

word_list = p.split(mystring)

print(word_list)


# In[ ]:





# In[50]:


import re

# defining string
mystring = "12and45, 78and85-17and89-97,54"

p = re.compile(r"and|[\s,-]+")

word_list = p.split(mystring)

print(word_list)


# In[ ]:





# ### b. The `re.Pattern.sub()` and `re.Pattern.subn()` Methods
# - - Python regex offers `sub()` the `subn()` methods to `search` and `replace` patterns in a string. Using these methods we can replace one or more occurrences of a regex pattern in the target string with a substitute string.
# 
# - The `sub()` method return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in `string` by the replacement `repl`.
# 
# **`p.sub(repl, string, count=0)`**
# 
# - Where,
#     - `repl`: The replacement that we are going to insert for each occurrence of a pattern. The replacement can be a string or function.
#     - `string`: The variable pointing to the target string (In which we want to perform the replacement).
#     - `count`: The default value of count is zero, means, find and replace all occurrences of pattern with replacement. For count=n, means replace first n occurrencesof pattern with the replacement
#     
#     
# 
# - It returns the string obtained by replacing the pattern occurrences in the string with the replacement string. If the pattern isn’t found, the string is returned unchanged.

# Replace white space with underscore character

# In[52]:


import re

# defining string
mystring = "Learning is fun with Arif Butt"

p = re.compile(r"\s")

word_list = p.sub("_", mystring)

print(word_list)


# In[ ]:





# Remove whitespaces from a string

# In[53]:


import re

# defining string
mystring = "Learning is fun with Arif Butt"

p = re.compile(r"\s+")

word_list = p.sub("", mystring)

print(word_list)


# In[ ]:





# Remove leading Spaces from a string

# In[54]:


import re

# defining string
mystring = "         Learning is fun with Arif Butt"

# ^\s+ remove only leading spaces
# caret (^) matches only at the start of the string
p = re.compile(r"^\s+")

word_list = p.sub("", mystring)

print(word_list)


# In[ ]:





# Remove both leading and trailing spaces

# In[55]:


import re

# defining string
mystring = "         Learning is fun with Arif Butt         \t. "

# ^\s+ remove leading spaces
# ^\s+$ removes trailing spaces
p = re.compile(r"^\s+|\s+$")

word_list = p.sub("", mystring)

print(word_list)


# In[60]:


string1 = 'a aa ab bc bb abc abcd ba aaaa'
p = re.compile("[abc]")
matches = p.finditer(string1)


# In[61]:


for m in matches:
    print(m)


# In[ ]: