#!/usr/bin/env python
# coding: utf-8

# # Regex introduction
# 
# ## What is a regex?
# [**Regex**](https://en.wikipedia.org/wiki/Regular_expression) stands for _regular expression_, and regular expressions are a way of writing patterns that match strings. Usually these patterns can be used to search strings for specific things, or to search and then replace certain things, etc. Regular expressions are great for string manipulation!
# 
# ## Why do regular expressions matter?
# From the first paragraph in this guide you might have guessed it, but regular expressions can be very useful **whenever you have to deal with strings**. From the basic renaming of a set of similarly named variables in your source code to [data preprocessing](https://github.com/clone95/Virgilio/blob/master/Specializations/HardSkills/DataPreprocessing.md). Regular expressions usually offer a concise way of expressing whatever type of things you want to find. For example, if you wanted to parse a form and look for the year that someone might have been born in, you could use something like `(19)|(20)[0-9][0-9]`. This is an example of a regular expression!
# 
# ## Prerequisites
# This guide does not assume any prior knowledge. Examples will be coded in Python, but mastery of the programming language is neither assumed nor needed. You are welcome to read the guide in your browser or to download it and to run the examples/toying around with them.
# 
# # Index
#  - [Basic regex](#Basic-regex)
#    - [Using Python re](#Using-Python-re)
#    - [$\pi$ lookup](#$\pi$-lookup)
#  - [Matching options](#Matching-options)
#    - [Virgilio or Virgil?](#Virgilio-or-Virgil?)
#  - [Matching repetitions](#Matching-repetitions)
#    - [Greed](#Greed)
#    - [Removing excessive spaces](#Removing-excessive-spaces)
#  - [Character classes](#Character-classes)
#    - [Phone numbers v1](#Phone-numbers-v1)
#  - [More `re` functions](#More-re-functions)
#    - [`search` with `match`](#search-with-match)
#    - [Count matches with `findall`](#Count-matches-with-findall)
#  - [Special characters](#Special-characters)
#    - [Phone numbers v2](#Phone-numbers-v2)
#  - [Groups](#Groups)
#    - [Phone numbers v3](#Phone-numbers-v3)
#  - [Toy project about regex](#Toy-project-about-regex)
#  - [Further reading](#Further-reading)
#  - [Suggested solutions](#Suggested-solutions)
#  
# Let's dive right in!

# **Just a quick word:** I tried to include some small exercises whenever I show you something new, so that you can try and test your knowledge. Examples of solutions are provided in the [end of the notebook](#Suggested-solutions).

# ## Basic regex
# 
# A regex is just a string written in a certain format, that can then be used by specific tools/libraries/programs to perform pattern matching on strings. Throughout this guide we will use `this formatting` to refer to regular expressions!
# 
# The simplest regular expressions that one can create are just composed of regular characters. If you wanted to find all the occurrences of the word _"Virgilio"_ in a text, you could write the regex `Virgilio`. In this regular expression, no character is doing anything special or different. In fact, this regular expression is just a normal word. That is ok, regular expressions are strings, after all!
# 
# If you were given the text _"Project Virgilio is great"_, you could use your `Virgilio` regex to find the occurrence of the word _"Virgilio"_. However, if the text was _"Project virgilio is great"_, then your regex wouldn't work, because regular expressions are **case-sensitive** by default and thus should match everything exactly. We say that `Virgilio` matches the sequence of characters "Virgilio" literally.

# ### Using Python re
# 
# To check if our regular expressions are working well and to give you the opportunity to directly experiment with them, we will be using Python's `re` module to work with regular expressions. To use the `re` module we first import it, then define a regular expression and then use the `search()` function over a string! Pretty simple:

# In[2]:


import re

regex = "Virgilio"
str1 = "Project Virgilio is great"
str2 = "Project virgilio is great"

if re.search(regex, str1):
    print("'{}' is in '{}'".format(regex, str1))
else:
    print("'{}' is not in '{}'".format(regex, str1))
    
if re.search(regex, str2):
    print("'{}' is in '{}'".format(regex, str2))
else:
    print("'{}' is not in '{}'".format(regex, str2))


# The `re.search(regex, string)` function takes a regex as first argument and then searches for any matches over the string that was given as the second argument. However, the return value of the function is **not** a boolean, but a *match object*:

# In[3]:


print(re.search(regex, str1))


# Match objects have relevant information about the match(es) encountered: the start and end positions, the string that was matched, and even some other things for more complex regular expressions.
# 
# We can see that in this case the match is exactly the same as the regular expression, so it may look like the `match` information inside the match object is irrelevant... but it becomes relevant as soon as we introduce options or repetitions into our regex.
# 
# If no matches are found, then the `.search()` function returns `None`:

# In[4]:


print(re.search(regex, str2))


# Whenever the match is not `None`, we can save the returned match object and use it to extract all the needed information!

# In[5]:


m = re.search(regex, str1)
if m is not None:
    print("The match started at pos {} and ended at pos {}".format(m.start(), m.end()))
    print("Or with tuple notation, the match is at {}".format(m.span()))
    print("And btw, the actual string matched was '{}'".format(m.group()))


# Now you should try to get some more matches and some fails with your own literal regular expressions. I provide three examples of my own:

# In[6]:


m1 = re.search("regex", "This guide is about regexes")
if m1 is not None:
    print("The match is at {}\n".format(m1.span()))

m2 = re.search("abc", "The alphabet goes 'abdefghij...'")
if m2 is None:
    print("Woops, did I just got the alphabet wrong..?\n")
    
s = "aaaaa aaaaaa a aaa"
m3 = re.search("a", s)
if m3 is not None:
    print("I just matched '{}' inside '{}'".format(m3.group(), s))


# ### $\pi$ lookup
# 
# $$\pi = 3.1415\cdots$$
# 
# right? Well, what comes after the dots? An infinite sequence of digits, right? Could it be that your date of birth appears in the first million digits of $\pi$? Well, we could use a regex to find that out! Change the `regex` variable below to look for your date of birth or for any number you want, in the first million digits of $\pi$!

# In[7]:


pifile = "regex-bin/pi.txt"
regex = ""  # define your regex to look your favourite number up

with open(pifile, "r") as f:
    pistr = f.read()  # pistr is a string that contains 1M digits of pi
    
## search for your number here


# To search for numbers in the first 100 million digits of $\pi$ (or 200 million, I didn't really get it) you can check [this](https://www.angio.net/pi/piquery) website.

# ## Matching options
# 
# We just saw a very simple regular expression that was trying to find the word _"Virgilio"_ in text, but we also saw that we had zero flexibility and we couldn't even handle the fact that someone may have forgotten to capitalize the name properly, spelling it like _"virgilio"_ instead.
# 
# To prevent problems like this, regular expressions can be written in a way to handle different possibilities. For our case, we want the first letter to be either _"V"_ or _"v"_, and that should be followed by _"irgilio"_.
# 
# In order to handle different possibilities, we use the character `|`. For instance, `V|v` matches the letter vee, regardless of its capitalization:

# In[8]:


v = "v"
V = "V"
regex = "v|V"
if re.search(regex, v):
    print("small v found")
if re.search(regex, V):
    print("big V found")


# Now we can concatenate the regex for the first letter and the `irgilio` regex (for the rest of the name) to get a regex that matches the name of Virgilio, regardless of the capitalization of its first letter:

# In[9]:


virgilio = "virgilio"
Virgilio = "Virgilio"
regex = "(V|v)irgilio"
if re.search(regex, virgilio):
    print("virgilio found!")
if re.search(regex, Virgilio):
    print("Virgilio found!")


# Notice that we write the regex with parenthesis: `(V|v)irgilio`
# 
# If we only wrote `V|virgilio`, then the regular expression would match either "V" or "virgilio", instead of "Virgilio" or "virgilio":

# In[3]:


regex = "V|virgilio"
print(re.search(regex, "This sentence only has a big V"))


# So we really need to parenthesize the `(V|v)` there. If we do, it will work as expected!

# In[4]:


regex = "(V|v)irgilio"
print(re.search(regex, "The name of the project is virgilio, but with a big V!"))
print(re.search(regex, "This sentence only has a big V"))


# Maybe you didn't even notice, but there is something else going on! Notice that we used the characteres `|`, `(` and `)`, and those are not present in the word _"virgilio"_, but nonetheless our regex `(V|v)irgilio` matched it... that is because these three characters have special meanings in the regex world, and hence are **not** interpreted literally, contrary to what happens to any letter in `irgilio`.

# ### Virgilio or Virgil?
# 
# Here is a couple of paragraphs from Wikipedia's [article on Virgil](https://en.wikipedia.org/wiki/Virgil):
# 
#  > Publius Vergilius Maro (Classical Latin: [ˈpuː.blɪ.ʊs wɛrˈɡɪ.lɪ.ʊs ˈma.roː]; traditional dates October 15, 70 BC – September 21, 19 BC[1]), usually called Virgil or Vergil (/ˈvɜːrdʒɪl/) in English, was an ancient Roman poet of the Augustan period. He wrote three of the most famous poems in Latin literature: the Eclogues (or Bucolics), the Georgics, and the epic Aeneid. A number of minor poems, collected in the Appendix Vergiliana, are sometimes attributed to him.[2][3]
# 
#  > Virgil is traditionally ranked as one of Rome's greatest poets. His Aeneid has been considered the national epic of ancient Rome since the time of its composition. Modeled after Homer's Iliad and Odyssey, the Aeneid follows the Trojan refugee Aeneas as he struggles to fulfill his destiny and reach Italy, where his descendants Romulus and Remus were to found the city of Rome. Virgil's work has had wide and deep influence on Western literature, most notably Dante's Divine Comedy, in which Virgil appears as Dante's guide through Hell and Purgatory.
#  
# "Virgilio" is the italian form of "Virgil", and I edited the above paragraphs to have the italian version instead of the english one. I want you to revert this!
# 
# You might want to take a look at [`while` cycles in Python](https://realpython.com/python-while-loop/), [string indexing](https://www.digitalocean.com/community/tutorials/how-to-index-and-slice-strings-in-python-3) and [string concatenation](https://realpython.com/python-string-split-concatenate-join/). The point is that you find a match, you break the string into the part _before_ the match and the part _after_ the match, and you glue those two together with _Virgilio_ in between.
# 
# Notice that [string replacement](https://www.tutorialspoint.com/python/string_replace.htm) would probably be faster and easier, but that would defeat the purpose of this exercise. After fixing everything, print the final results to be sure that you fixed every occurrence of the name.

# In[11]:


paragraphs = \
"""Publius Vergilius Maro (Classical Latin: [ˈpuː.blɪ.ʊs wɛrˈɡɪ.lɪ.ʊs ˈma.roː]; traditional dates October 15, 70 BC – September 21, 19 BC[1]), usually called virgilio or Vergil (/ˈvɜːrdʒɪl/) in English, was an ancient Roman poet of the Augustan period. He wrote three of the most famous poems in Latin literature: the Eclogues (or Bucolics), the Georgics, and the epic Aeneid. A number of minor poems, collected in the Appendix Vergiliana, are sometimes attributed to him.[2][3]

Virgilio is traditionally ranked as one of Rome's greatest poets. His Aeneid has been considered the national epic of ancient Rome since the time of its composition. Modeled after Homer's Iliad and Odyssey, the Aeneid follows the Trojan refugee Aeneas as he struggles to fulfill his destiny and reach Italy, where his descendants Romulus and Remus were to found the city of Rome. virgilio's work has had wide and deep influence on Western literature, most notably Dante's Divine Comedy, in which virgilio appears as Dante's guide through Hell and Purgatory."""


# ## Matching repetitions
# 
# Sometimes we want to find patterns that have bits that will be repeated. For example, people make a _"awww"_ or _"owww"_ sound when they see something cute, like a baby. But the number of _"w"_ I used there was completely arbitrary! If the baby is really really cute, someone might write _"awwwwwwwwwww"_. So how can I write a regex that matches _"aww"_ and _"oww"_, but with an arbitrary number of characters _"w"_?
# 
# I will illustrate several ways of capturing repetitions, by testing regular expressions against the following strings:
# 
#  - "awww" (3 letters "w")
#  - "awwww" (4 letters "w")
#  - "awwwwwww" (7 letters "w")
#  - "awwwwwwwwwwwwwwww" (16 letters "w")
#  - "aw" (1 letter "w")
#  - "a" (0 letters "w")

# In[12]:


cute_strings = [
    "awww",
    "awwww",
    "awwwwwww",
    "awwwwwwwwwwwwwwww",
    "aw",
    "a"
]

def match_cute_strings(regex):
    """Takes a regex, prints matches and non-matches"""
    for s in cute_strings:
        m = re.search(regex, s)
        if m:
            print("match: {}".format(s))
        else:
            print("non match: {}".format(s))


# #### At least once
# 
# If I want to match all strings that containt **at least** one "w", we can use the character `+`. A `+` means that we want to find **one or more repetitions** of whatever was to the left of it. For example, the regex `a+` will match any string that has at least one "a".

# In[13]:


regex = "aw+"
match_cute_strings(regex)


# #### Any number of times
# 
# If I want to match all strings that contain an arbitrary number of letters "w", I can use the character `*`. The character `*` means **match any number of repetitions** of whatever comes on the left of it, _even 0 repetitions_! So the regex `a*` would match the empty string "", because the empty string "" has 0 repetitions of the letter "a".

# In[14]:


regex = "aw*"
match_cute_strings(regex)


# #### A specific number of times
# 
# If I want to match a string that contains a certain particle a specific number of times, I can use the `{n}` notation, where `n` is replaced by the number of repetitions I want. For example, `a{3}` matches the string "aaa" but not the string "aa".

# In[15]:


regex = "aw{3}"
match_cute_strings(regex)


# **Wait a minute**, why did the pattern `aw{3}` match the longer expressions of cuteness, like "awwww" or "awwwwwww"? Because the regular expressions try to find _substrings_ that match the pattern. Our pattern is `awww` (if I write the `w{3}` explicitly) and the string **awww**w has that substring, just like the string **awww**wwww has it, or the longer version with 16 letters "w". If we wanted to exclude the strings "awwww", "awwwwwww" and "awwwwwwwwwwwwwwww" we would have to fix our regex. A better example that demonstrates how `{n}` works is by considering, instead of expressions of cuteness, expressions of amusement like "wow", "woow" and "wooooooooooooow". We define some expressions of amusement:
# 
#  - "wow"
#  - "woow"
#  - "wooow"
#  - "woooow"
#  - "wooooooooow"
#  
# and now we test our `{3}` pattern.

# In[16]:


wow_strings = [
    "wow",
    "woow",
    "wooow",
    "woooow",
    "wooooooooow"
]

def match_wow_strings(regex):
    """Takes a regex, prints matches and non-matches"""
    for s in wow_strings:
        m = re.search(regex, s)
        if m:
            print("match: {}".format(s))
        else:
            print("non match: {}".format(s))


# In[17]:


regex = "wo{3}w"
match_wow_strings(regex)


# #### Between $n$ and $m$ times
# 
# Expressing amusement with only three "o" is ok, but people might also use two or four "o". How can we capture a variable number of letters, but within a range? Say I only want to capture versions of "wow" that have between 2 and 4 letters "o". I can do it with `{2,4}`.

# In[18]:


regex = "wo{2,4}w"
match_wow_strings(regex)


# #### Up to $n$ times or at least $m$ times
# 
# Now we are just playing with the type of repetitions we might want, but of course we might say that we want **no more** than $n$ repetitions, which you would do with `{,n}`, or that we want **at least** $m$ repetitions, which you would do with `{m,}`.
# 
# In fact, take a look at these regular expressions:

# In[19]:


regex = "wo{,4}w" # should not match strings with more than 4 o's
match_wow_strings(regex)


# In[20]:


regex = "wo{3,}w" # should not match strings with less than 3 o's
match_wow_strings(regex)


# #### To be or not to be
# 
# Last but not least, sometimes we care about something that might or might not be present. For example, above we dealed with the English and Italian versions of the name Virgilio. If we wanted to write a regular expression to capture both versions, we could write `((V|v)irgil)|((V|v)irgilio)`, or slightly more compact, `(V|v)((irgil)|(irgilio))`. But this does not look good at all, right? All we need to say is that the final "io" might or might not be present. We do this with the `?` character. So the regex `(V|v)irgil(io)?` matches the upper and lower case versions of "Virgil" and "Virgilio".

# In[21]:


regex = "(V|v)irgil(io)?"
names = ["virgil", "Virgil", "virgilio", "Virgilio"]
for name in names:
    m = re.search(regex, name)
    if m:
        print("The name {} was matched!".format(name))


# ### Greed
# 
# The `+`, `?`, `*` and `{,}` operators are all greedy. What does this mean? It means that they will try to match as much as possible. They have this default behaviour, as opposed to stopping to try and find more matches as soon as the regex is satisfied. To better illustrate what I mean by this, let us look again at the information contained in the `match` object we have been dealing with:

# In[26]:


regex = "a+"
s = "aaa"
m = re.search(regex, s)
print(m)


# Notice the part of the printed information that says `match='aaa'`. The function `m.group()` will let me know what was the actual string that was matched by the regular expression, and in this case it was "aaa". Why does it make sense to have access to this information? Well, the regex I wrote, `a+`, will match one or more letters "a" in a row. If I use the regex over a string and I get a match, how would I be able to know how many "a"s were matched, if I didn't have access to that type of information?

# In[29]:


print(m.group())


# So let us verify that, in fact, the operators I mentioned are all greedy. Again, because they all match as many characters as they can.
# 
# Below, we see that given a string of thirty times the letter "a",
# 
#   - the pattern `a?` matches 1 "a", which is as much as it could
#   - the pattern `a+` matches 30 "a"s, which is as much as it could
#   - the pattern `a*` also matches 30
#   - the pattern `a{5,10}` matches 10 "a"s, which was the limit imposed by us

# In[31]:


s = "a"*30
print(re.search("a?", s).group())
print(re.search("a+", s).group())
print(re.search("a*", s).group())
print(re.search("a{5,10}", s).group())


# If we don't want our operators to be greedy, we just put an extra `?` after them. So the following regular expressions are **not** greedy:
# 
#   - the pattern `a??` will match **no** characters, much like `a*?`, because now their goal is to match as little as possible. But a match of length 0 is the shortest match possible!
#   - the pattern `a+?` will only match 1 "a"
#   - the pattern `a{5,10}?` will only match 5 "a"s
#   
# We can easily confirm what I just said by running the code below. Notice that now I print things differently, because otherwise we wouldn't be able to see the `a??` and `a*?` patterns matching nothing.

# In[34]:


s = "a"*30
print("'{}'".format(re.search("a??", s).group()))
print("'{}'".format(re.search("a+?", s).group()))
print("'{}'".format(re.search("a*?", s).group()))
print("'{}'".format(re.search("a{5,10}?", s).group()))


# ### Removing excessive spaces
# 
# Now that we know about repetitions, I am going to tell you about the `sub` function and we are going to use that to parse a piece of text and remove all extra spaces that are present. Typing in `re.sub(regex, rep, string)` will use the given regex on the given string, and whenever it matches, it removes the match and puts the `rep` in there.
# 
# For example, I can use that to replace all English/Italian occurrences of the name Virgilio with a standardized one:

# In[39]:


s = "Virgilio has many names, like virgil, virgilio, Virgil, Vergil, or even vergil."
regex = "(V|v)(e|i)rgil(io)?"

print(
    re.sub(regex, "Virgilio", s)
)


#     Now   it  is your   turn.  I am     going  to give   you this    sentence as        input, and   your  job    is to      fix the     whitespace         in it. When you    are  done,    save the    result in a  string  named   `s`, and   check    if  `s.count("  ")` is   equal   to    0  or not.

# In[ ]:


weird_text = "Now   it  is your   turn.  I am     going  to give   you this    sentence as        input, and   your  job    is to      fix the     whitespace         in it. When you    are  done,    save the    result in a  string  named   `s`, and   check    if  `s.count("  ")` is   equal   to    0  or not."
regex = ""  # put your regex here

# substitute the extra whitespace here
# save the result in 's'

# this print should be 0
print(s.count("  "))


# ## Character classes
# 
# So far we have been using writing some simple regular expressions that have been matching some words, and some names, and things like that. Now we have a different plan. We will write a regular expression that will match on US phone numbers, which we will assume are of the form xxx-xxx-xxxx. The first three digits are the area code, but we will not care about whether the area code actually makes sense or not. How do we match this, then?
# 
# In fact, how can I match the first digit? It can be any number from 0 to 9, so should I write `(0|1|2|3|4|5|6|7|8|9)` to match the first digit, and then repeat? Actually, we could do that, yes, to get this regex:
# 
# `(0|1|2|3|4|5|6|7|8|9){3}-(0|1|2|3|4|5|6|7|8|9){3}-(0|1|2|3|4|5|6|7|8|9){4}`
# 
# Does this work?

# In[45]:


regex = "(0|1|2|3|4|5|6|7|8|9){3}-(0|1|2|3|4|5|6|7|8|9){3}-(0|1|2|3|4|5|6|7|8|9){4}"
numbers = [
    "202-555-0181",
    "202555-0181",
    "202 555 0181",
    "512-555-0191",
    "96-125-3546",
]
for nr in numbers:
    print(re.search(regex, nr))


# It looks like it works, but surely there must be a better way... and there is! Instead of writing out every digit like we did, we can actually write a range of values! In fact, the regex `[0-9]` matches all digits from 0 to 9. So we can actually shorten our regex to `[0-9]{3}-[0-9]{3}-[0-9]{4}`:

# In[46]:


regex = "[0-9]{3}-[0-9]{3}-[0-9]{4}"
numbers = [
    "202-555-0181",
    "202555-0181",
    "202 555 0181",
    "512-555-0191",
    "96-125-3546",
]
for nr in numbers:
    print(re.search(regex, nr))


# The magic here is being done by the `[]`, which denotes a character class. The way `[]` works is, the regex will try to match any of the things that are inside, and it just so happens that `0-9` is a shorter way of listing all the digits. Of course you could also do `[0123456789]{3}-[0123456789]{3}-[0123456789]{4}` which is slightly shorter than our first attempt, but still pretty bad. Similar to `0-9`, we have `a-z` and `A-Z`, which go through all letters of the alphabet.
# 
# You can also start and end in different places, for example `c-o` can be used to match words that only use letters between the "c" and the "o", like "hello":

# In[47]:


regex = "[c-o]+"
print(re.search(regex, "hello"))
print(re.search(regex, "rice"))


# With these character classes we can actually rewrite our Virgilio regex into something slightly shorter, going from `(V|v)(e|i)rgil(io)?` to `[Vv][ie]rgil(io)?`.

# In[53]:


s = "Virgilio has many names, like virgil, virgilio, Virgil, Vergil, or even vergil."
regex = "[Vv][ie]rgil(io)?"

print(
    re.sub(regex, "Virgilio", s)
)


# Again we see that our regular expression matched the **ice** in r**ice**, because the "r" was not inside the legal range of letters, but **ice** was.
# 
# The _character class_ is the square brackets `[]` and whatever goes inside it. Also, note that the special characters we have been using lose their meaning inside a character class! So `[()?+*{}]` will actually look to match any of those characters:

# In[54]:


regex = "[()?+*{}]"
print(re.search(regex, "Did I just ask a question?"))


# A final note on character classes, if they start with `^` then we are actually saying "use everything _except_ what is inside this":

# In[55]:


regex = "[^c-o]+"
print(re.search(regex, "hello"))
print(re.search(regex, "rice"))


# ### Phone numbers v1
# 
# Now that you know how to use character classes to denote ranges, you need to write a regular expression that matches american phone numbers with the format xxx-xxx-xxxx. Not only that, but you must also cope with the fact that the numbers may or may not be preceeded by the country indicator, which you can assume that will look like "+1" or "001". The country indicator may be separated from the rest of the number with a space or with a dash.

# In[ ]:


regex = ""  # write your regex here
matches = [  # you should be able to match those
    "202-555-0181",
    "001 202-555-0181",
    "+1-512-555-0191"
]
non_matches = [  # for now, none of these should be matched
    "202555-0181",
    "96-125-3546",
    "(+1)5125550191"
]
for s in matches:
    print(re.search(regex, s))
for s in non_matches:
    print(re.search(regex, s))


# ## More `re` functions
# 
# So far we only looked at the `.search()` function of the `re` module, but now I am going to tell you about a couple more function that can be quite handy when you are dealing with pattern matching. By the time you are done with this small section, you will now the following functions: `match()`, `search()`, `findall()`, `sub()` and `split()`.
# 
# If you are here mostly for the regular expressions, and you don't care much about using them with Python, you can just skim through this section... even though it is still a nice read.

# #### `search()` and `sub()`
# 
# You already know these two functions, `re.search(regex, string)` will try to find your pattern given by `regex` in the given `string` and return the information of the match in a `match` object. The function `re.sub(regex, rep, string)` will take a regex and two strings; it will then look for the pattern you specified in `string` and replace the matches with the other string `rep` you gave it.

# #### `match()`
# 
# The function `re.match(regex, string)` is similar to the function `re.search()`, except that `.match()` will only check if your pattern applies to the **beginning** of the string. That is, if your string does not **start** with the pattern you provided, the function returns `None`.

# In[4]:


regex = "abc"
string1 = "abcdef"
string2 = "the alphabet starts with abc"
# the .search() function finds the patterns, regardless of position
if re.search(regex, string1):
    print(".search() found {} in {}".format(regex, string1))
if re.search(regex, string2):
    print(".search() found {} in {}".format(regex, string2))
    
# the .match() function only checks if the string STARTS with the pattern
if re.match(regex, string1):
    print(".match() says that {} starts with {}".format(string1, regex))
if re.match(regex, string2):  # this one should NOT print
    print(".match() says that {} starts with {}".format(string2, regex))


# #### `findall()`
# 
# The `re.findall(regex, string)` is exactly like the `.search()` function, except that it will return **all** the matches it can find, instead of just the first one. Instead of returning a `match` object, it just returns the string that matched.

# In[5]:


regex = "wow"
string = "wow wow wow!"

print(re.search(regex, string))

print(re.findall(regex, string))


# In[6]:


regex = "ab[0-9]"
string = "ab1 ab2 ab3"

print(re.search(regex, string))

print(re.findall(regex, string))


# It is important to note that the `findall()` function only returns _non-overlaping_ matches. That is, one could argue that `wow` appears twice in "wowow", in the beginning: **wow**ow, and in the end: wo**wow**. Nonetheless, `findall()` only returns one match because the second match overlaps with the first:

# In[7]:


regex = "wow"
string = "wowow"
print(re.findall(regex, string))


# With this information it now makes a bit more sense to consider the greediness of the operators we showed before, like `?` and `+`. Imagine we are dealing with the regex `a+` and we have a string "aaaaaaaaa". If we use the greedy version of `+`, then we get a single match which is the whole string. If we use the non-greedy version of the operator `+`, perhaps because we want as many matches as possible, we will get a bunch of "a" matches!

# In[8]:


regex_greedy = "a+"
regex_nongreedy = "a+?"
string = "aaaaaaaaa"

print(re.findall(regex_greedy, string))

print(re.findall(regex_nongreedy, string))


# #### `split()`
# 
# The `re.split(regex, string)` splits the given string into bits wherever it is able to find the pattern you specified. Say we are interested in finding all the sequences of consecutive consonants in a sentence (I don't know why you would want that...). Then we can use the vowels and the space " " to break up the sentence:

# In[11]:


regex = "[aeiou ]+" # this will eliminate all vowels/spaces that appear consecutively
string = "This is just a regular sentence"

print(re.split(regex, string))


# ### `search` with `match`
# 
# Recall that the `match()` function only checks if your pattern is in the beginning of the string. What I want you to do is define your own `search` function that takes a regex and a string, and returns `True` if the pattern is inside the string, and `False` otherwise. Can you do it?

# In[15]:


def my_search(regex, string):
    pass  # write your code here

regex = "[0-9]{2,4}"

# your function should be able to match in all these strings
string1 = "1984 was already some years ago."
string2 = "There is also a book whose title is '1984', but the story isn't set in the year of 1984."
string3 = "Sometimes people write '84 for short."

# your function should also match with this regex and this string
regex = "a*"
string = ""


# ### Count matches with `findall`
# 
# Now I want you to define the `count_matches` function, which takes a regex and a string, and returns the number of non-overlaping matches there exist in the given string. Can you do it?

# In[25]:


def count_matches(regex, string):
    pass  # your code goes here

regex = "wow"

string1 = "wow wow wow" # this should be 3
string2 = "wowow" # this should be 1
string3 = "wowowow" # this should be 2


# ## Special characters
# 
# It is time to ramp things up a bit! We have seen some characters that have special meanings, and now I am going to introduce a couple more of those! I will start by listing them, and then I'll explain them in more detail:
# 
#  - `.` is used to match **any** character, except for a newline
#  - `^` is used to match at the beginning of the string
#  - `$` is used to match at the end of the string
#  - `\d` is used to match any digit
#  - `\w` is used to match any alphanumeric character
#  - `\s` is used to match any type of whitespace
#  - `\` is used to remove the special meaning of the characters

# #### Dot `.`
# 
# The `.` can be used in a regular expression to capture any character that might have been used there, as long as we are still in the same line. That is, the only place where `.` doesn't work is if we changed lines in the text. Imagine the pattern was `d.ck`. Then the pattern would match
# 
# ```
# "duck"```
# 
# but it would not match
# 
# ```
# "d
# ck"```
# 
# because we changed lines in the middle of the string.

# #### Caret `^`
# 
# If we use a `^` in the beginning of the regular expression, then we only care about matches in the beginning of the string. That is, `^wow` would only match if the string started with "wow":

# In[30]:


regex = "^wow"

print(re.search(regex, "wow, this is awesome"))
print(re.search(regex, "this is awesome, wow"))


# Recall that `^` inside the character class can also mean "anything but whatever is in this class", so the regular expression `[^d]uck` would match any string that has **uck** in it, as long as it is not the word "duck". If the caret `^` appears inside a character class `[]` but it is not the first character, than it has no special meaning and it just stands for the character itself. This means that the regex `[()^{}]` is looking to match any of the characters listed:

# In[29]:


regex = "[()^{}]"
print(re.search(regex, "^"))
print(re.search(regex, "("))
print(re.search(regex, "}"))


# #### Dollar sign `$`
# 
# Contrary to the caret `^`, the dollar sign only matches at the end of the string!

# In[31]:


regex = "wow$"

print(re.search(regex, "wow, this is awesome"))
print(re.search(regex, "this is awesome, wow"))


# Combining the `^` with the `$` means we are looking to match the whole string with our pattern. For example `^[a-zA-Z ]*$` checks if our string only contains letters and spaces and nothing else:

# In[32]:


regex = "^[a-zA-Z ]*$"

s1 = "this is a sentence with only letters and spaces"
s2 = "this sentence has 1 number"
s3 = "this one has punctuation..."

print(re.search(regex, s1))
print(re.search(regex, s2))
print(re.search(regex, s3))


# #### Character groups `\d`, `\w` and `\s`
# 
# Whenever you see a backslash followed by a letter, that probably means that something _special_ is going on. These three special "characters" are shorthand notation for some character classes `[]`. For example, the `\d` is the same as `[0-9]`. The `\w` represents any alphanumeric character (like letters, numbers and `_`), and `\s` represents any whitespace character (like the space " ", the tab, the newline, etc).
# 
# All these three special characters I showed, can be capitalized. If they are, then they mean the exact opposite! So `\D` means "anything **except** a digit", `\W` means "anything **except** an alphanumeric character" and `\S` means "anything **except** whitespace characters.

# In[43]:


regex = "\D+"
s = "these are some words"
print(re.findall(regex, s))


# Adding up to that, these special characters can be used inside a character class, so for instance `[abc\d]` would match any digit and the letters "a", "b" and "c". If the caret character `^` is used, then we are excluding whatever the special character refers to. As an example, if `[\d]` would match any digit, then `[^\d]` will match anything that is not a digit.

# #### The backslash `\`
# 
# We already saw the backslash being used before letters to give them some special meaning... Well, the backslash before a special character also strips it of its special meaning! So, if you wanted to match a backslash, you could use `\\`. If you want to match any of the other special characters we already saw, you could put a `\` before them, like `\+` to match a plus sign. The next regular expression can be used to match an addition expression like "16 + 6"

# In[45]:


regex = "[\d]+ ?\+ ?[\d]+"
add1 = "16 + 6"
add2 = "4325+2"
add3 = "4+ 564"
mult1 = "56 * 2"

print(re.search(regex, add1))
print(re.search(regex, add2))
print(re.search(regex, add3))
print(re.search(regex, mult1))


# ### Phone numbers v2
# 
# Now I invite you to take a look at [Phone numbers v1](#Phone-numbers-v1) and rewrite your regular expression to include some new special characters that you didn't know before!

# In[ ]:


regex = ""  # write your regex here
matches = [  # you should be able to match those
    "202-555-0181",
    "001 202-555-0181",
    "+1-512-555-0191"
]
non_matches = [  # for now, none of these should be matched
    "202555-0181",
    "96-125-3546",
    "(+1)5125550191"
]
for s in matches:
    print(re.search(regex, s))
for s in non_matches:
    print(re.search(regex, s))


# ## Groups
# 
# So far, when we used a regex to match a string we could retrieve the whole information of the match by using the `.group()` function on the match object:

# In[5]:


regex = "my name? is"

m = re.search(regex, "my nam is Virgilio")
if m is not None:
    print(m.group())


# Say we are dealing with phone numbers again, and we want to look for phone numbers in a big text. But after that, we also want to extract the country from where the number is from. How could we do it..? Well, we can use a regex to match the phone numbers, and then use a second regex to extract the country code, right? (Let us just assume that phone numbers are written with the digits all in a sequence, with no spaces or "-" separating them.)

# In[13]:


regex_number = "((00|[+])\d{1,3}[ -])\d{8,12}"
regex_code = "((00|[+])\d{1,3})"
matches = [  # you should be able to match those
    "+351 2025550181",
    "001 2025550181",
    "+1-5125550191",
    "0048 123456789"
]

for s in matches:
    m = re.search(regex_number, s)  # match the phone number
    if m is not None:
        phone_number = m.group()    # extract the phone number
        code = re.search(regex_code, phone_number)  # match the country code
        print("The country code is: {}".format(code.group()))


# But not only is this repetitive, because I just copied the beginning of the `regex_number` into the `regex_code`, but it becomes very cumbersome if I am trying to retrieve several different parts of my match. Because of this, there is a functionality of regular expressions that is _grouping_. By grouping parts of the regular expression, you can do things like using the repetition operators on them and **retrieve their information** later on.
# 
# To do grouping, one only needs to use the `()` parenthesis. For example, the regex `(ab)+` looks for matches of the form "ab", "abab", "ababab", etcetera.
# 
# We also used the grouping [in the beginning](#Matching-options) to create a regex that matched "Virgilio" and "virgilio", by writing `(V|v)irgilio`.
# 
# Now off to the part that really matters! We can use grouping to retrieve portions of the matches, and we do that with the `.group()` function! Any set of `()` defines a group, and then we can use the `.group(i)` function to retrieve group `i`. Just note that the 0th group is always the whole match, and then you start counting from the left!

# In[14]:


regex_with_grouping = "(abc) (de(fg)hi)"
m = re.search(regex_with_grouping, "abc defghi jklm n opq")
print(m.group())
print(m.group(0))
print(m.group(1))
print(m.group(2))
print(m.group(3))
print(m.groups())


# Notice that `match.group()` and `match.group(0)` are the same thing. Also note that the function `match.groups()` returns all the groups in a tuple!

# ### Phone numbers v3
# 
# Using what you learned so far, write a regex that matches phone numbers with different country codes. Assume the following:
# 
#   - The country code starts with either `00` or `+`, followed by one to three digits
#   - The phone number has length between 8 and 12
#   - The phone number and country code are separated by a space " " or by a hyphen "-"
#   
# Have your code look for phone numbers in the string I will provide next, and have it print the different country codes it finds.
# 
# You might want to read what the exact behaviour of `re.findall()` is when the regex has groups in it. You can do that by checking the [documentation of the `re` module](https://docs.python.org/3/library/re.html#re.findall).

# In[15]:


paragraph = """Hello, I am Virgilio and I am from Italy.
If phones were a thing when I was alive, my number would've probably been 0039 3123456789.
I would also love to get a house with 3 floors and something like +1 000 square meters.
Now that we are at it, I can also tell you that the number 0039 3135313531 would have suited Leo da Vinci very well...
And come to think of it, someone told me that Socrates had dibs on +30-2111112222"""
# you should find 3 phone numbers
# and you should not be fooled by the other numbers that show up in the text


# ## Toy project about regex
# 
# For the toy project, that is far from trivial, you are left with mimicking what [I did here](http://mathspp.blogspot.com/2017/11/on-computing-all-patterns-matched-by.html). If you follow that link, you will find a piece of code that takes a regular expression and then prints all the strings that the given regex would match.
# 
# I'll just give you a couple of examples on how this works:

# In[ ]:


import sys
sys.path.append("./regex-bin")
import regexPrinter

def get_iter(regex):
    return regexPrinter.printRegex(regex).print()

def printall(regex):
    for poss_match in get_iter(regex):
        print(poss_match)

regex = "V|virgilio"
printall(regex)
print("-"*30)
regex = "wo+w"
printall(regex)
print("-"*30)
# notice that for some reason, dumb me used {n:m} instead of {n,m}
# also note that I only implemented {n,m}, and not {n,} nor {,m} nor {n}
# also note that this does not support nor \d nor [0-9]
regex = "((00|[+])1[ -])?[0123456789]{3:3}"
printall(regex)


# Note that the code is protected against infinite patterns, which are signaled with `...`.

# In[47]:


printall("this is infinite!+")


# If you are completely new to this sort of things, then this will look completely impossible... but it is not, because I am a normal person and I was able to do it! So if you really want you can also do it! In the link you have listed all the functionality I decided to include, which excluded `\d`, for example.
# 
# I was only able to do this in the way I did because I had gone through some (not all) of the blog posts in [this amazing series](https://ruslanspivak.com/lsbasi-part1/).
# 
# Maybe you can implement a smaller subset of the features without too much trouble? The point of this is that you could only print the strings matched by a regex if you know how regular expressions work. Try starting with only implementing literal matching and the `|` and `?` operators. Can you now include grouping `()` so that `(ab)?` would work as expected? Can you add `[]`? What about `+` and `*`? Or maybe start with `{n,m}` and write `?`, `+` and `*` as `{0,1}`, `{1,}` and `{0,}` respectively.
# 
# You can also postpone this project for a bit, and dig deeper into the world of regex. The next section contains some additional references and some websites with exercises to practice your new knowledge!

# ## Further reading
# For regular expressions in Python, you can take a look at the [documentation](https://docs.python.org/3/library/re.html) of the `re` module, as well as this [regex HOWTO](https://docs.python.org/3/howto/regex.html).
# 
# Some nice topics to follow up on this would include, but are not limited to:
#   - Non capturing groups (and named groups for Python)
#   - Lookaheads (positive, negative, ...)
#   - Regex compilation and flags (for Python)
#   - Recursive regular expressions
# 
# [This](https://regexr.com/) interesting website (and [this one](https://regex101.com/) as well) provides an interface for you to type regular expressions and see what they match in a text. The tool also gives you an explanation of what your regular expression is doing.
# 
# ---
# 
# I found some interesting websites with exercises on regular expressions. [This one](https://regexone.com/lesson/introduction_abcs) has more "basic" exercises, each one of them preceeded by an explanation of whatever you will need to complete the exercise. I suggest you to go through them. [Hackerrank](https://www.hackerrank.com/domains/regex) and [regexplay](http://play.inginf.units.it/#/) also have some interesting exercises, but those require you to login in some way.
# 
# ---
# 
# If you enjoyed this guide and/or it was useful, consider leaving a star in the [Virgilio repository](https://github.com/clone95/Virgilio) and sharing it with your friends!
# 
# This was brought to you by the editor of the [Mathspp Blog](https://mathspp.blogspot.com), [RojerGS](https://github.com/RojerGS).

# ### Suggested solutions

# ### $\pi$ lookup (solved)

# In[23]:


pifile = "regex-bin/pi.txt"
regex = "9876"  # define your regex to look your favourite number up

with open(pifile, "r") as f:
    pistr = f.read()  # pistr is a string that contains 1M digits of pi
    
## search for your number here
m = re.search(regex, pistr)
if m:
    print("Found the number '{}' at positions {}".format(regex, m.span()))
else:
    print("Sorry, the first million digits of pi can't help you with that...")


# ### Virgilio or Virgil? (solved)

# In[20]:


paragraphs = \
"""Publius Vergilius Maro (Classical Latin: [ˈpuː.blɪ.ʊs wɛrˈɡɪ.lɪ.ʊs ˈma.roː]; traditional dates October 15, 70 BC – September 21, 19 BC[1]), usually called virgilio or Vergil (/ˈvɜːrdʒɪl/) in English, was an ancient Roman poet of the Augustan period. He wrote three of the most famous poems in Latin literature: the Eclogues (or Bucolics), the Georgics, and the epic Aeneid. A number of minor poems, collected in the Appendix Vergiliana, are sometimes attributed to him.[2][3]

Virgilio is traditionally ranked as one of Rome's greatest poets. His Aeneid has been considered the national epic of ancient Rome since the time of its composition. Modeled after Homer's Iliad and Odyssey, the Aeneid follows the Trojan refugee Aeneas as he struggles to fulfill his destiny and reach Italy, where his descendants Romulus and Remus were to found the city of Rome. virgilio's work has had wide and deep influence on Western literature, most notably Dante's Divine Comedy, in which virgilio appears as Dante's guide through Hell and Purgatory."""

regex = "(V|v)irgilio"
parsed_str = paragraphs
m = re.search(regex, parsed_str)
while m is not None:
    parsed_str = parsed_str[:m.start()] + "Virgil" + parsed_str[m.end():]
    m = re.search(regex, parsed_str)

print(parsed_str)


# ### Removing excessive spaces (solved)

# In[21]:


weird_text = "Now   it  is your   turn.  I am     going  to give   you this    sentence as        input, and   your  job    is to      fix the     whitespace         in it. When you    are  done,    save the    result in a  string  named   `s`, and   check    if  `s.count("  ")` is   equal   to    0  or not."
regex = " +"  # put your regex here
# there are several possible solutions, I chose this one

# substitute the extra whitespace here
s = re.sub(regex, " ", weird_text)

# this print should be 0
print(s.count("  "))
print(s)


# ### Phone numbers v1 (solved)

# In[48]:


regex = "((00|[+])1[ -])?[0-9]{3}-[0-9]{3}-[0-9]{4}"  # write your regex here
matches = [  # you should be able to match those
    "202-555-0181",
    "001 202-555-0181",
    "+1-512-555-0191"
]
non_matches = [  # for now, none of these should be matched
    "202555-0181",
    "96-125-3546",
    "(+1)5125550191"
]
for s in matches:
    print(re.search(regex, s))
for s in non_matches:
    print(re.search(regex, s))


# ### `search` with `matched` (solved)

# In[23]:


def my_search(regex, string):
    found = False
    while string:
        m = re.match(regex, string)
        if m:
            return True
        string = string[1:]
    # check if the pattern matches the empty string
    if re.match(regex, string):
        return True
    else:
        return False

regex = "[0-9]{2,4}"

# your function should be able to match in all these strings
string1 = "1984 was already some years ago."
print(my_search(regex, string1))
string2 = "There is also a book whose title is '1984', but the story isn't set in the year of 1984."
print(my_search(regex, string2))
string3 = "Sometimes people write '84 for short."
print(my_search(regex, string3))

# your function should also match with this regex and this string
regex = "a*"
string = ""
print(my_search(regex, string))


# ### Count matches with `findall` (solved)

# In[24]:


def count_matches(regex, string):
    return len(re.findall(regex, string))

regex = "wow"

string1 = "wow wow wow" # this should be 3
print(count_matches(regex, string1))
string2 = "wowow" # this should be 1
print(count_matches(regex, string2))
string3 = "wowowow" # this should be 2
print(count_matches(regex, string3))


# ### Phone numbers v2 (solved)

# In[49]:


regex = "((00|[+])1[ -])?\d{3}-\d{3}-\d{4}"  # write your regex here
matches = [  # you should be able to match those
    "202-555-0181",
    "001 202-555-0181",
    "+1-512-555-0191"
]
non_matches = [  # for now, none of these should be matched
    "202555-0181",
    "96-125-3546",
    "(+1)5125550191"
]
for s in matches:
    print(re.search(regex, s))
for s in non_matches:
    print(re.search(regex, s))


# #### Phone numbers v3 (solved)
# 
# For this "problem", one thinks of using the `.findall()` function to look for all matches. When we do that, we don't get a list of the match objects, but instead a list with tuples, where each tuple has a specific group from our regex. This is the behaviour that is [documented for the `re.findall()` function](https://docs.python.org/3/library/re.html#re.findall).
# 
# This is fine, because we really only cared about the number code, and we can print it easily. If we wanted the match objects, then the alternative would be to use the [`re.finditer()`](https://docs.python.org/3/library/re.html#re.finditer) function.

# In[26]:


paragraph = """Hello, I am Virgilio and I am from Italy.
If phones were a thing when I was alive, my number would've probably been 0039 3123456789.
I would also love to get a house with 3 floors and something like +1 000 square meters.
Now that we are at it, I can also tell you that the number 0039 3135313531 would have suited Leo da Vinci very well...
And come to think of it, someone told me that Socrates had dibs on +30-2111112222"""
# you should find 3 phone numbers
# and you should not be fooled by the other numbers that show up in the text

regex = "((00|[+])\d{1,3})[ -]\d{8,12}"
ns = re.findall(regex, paragraph)  # find numbers
for n in ns:
    # n is a tuple with the two groups our string has
    print(n)
    
for n in re.finditer(regex, paragraph):
    print("The number '{}' has country code: {}".format(n.group(), n.group(1)))


# In[ ]: