re
Modulere.compile()
Methodre.Pattern.search()
Methodre.Pattern.match()
Methodre.Pattern.findall()
Methodre.Pattern.finditer()
Methodre.Pattern.split()
Methodre.Pattern.sub()
Methodre.Pattern.subn()
MethodSpecial characters are characters that do not match themselves as seen but have a special meaning when used in a regular expression. Some commonly used wild cards or meta characters are listed below:
Wild Card | Description |
---|---|
^ | Caret symbol specifies that the match must start at the beginning of the string, and in MULTILINE mode also matches immediately after each newline - ^b will check if the string starts with 'b' such as baba, boss, basic, b, by, etc.- ^si will check if the string starts with 'si' such as simple, sister, si, etc. |
$ | Specifies that the match must occur at the end of the string - s$ will check for the string that ends with a such as geeks, ends, s, etc.- ing$ will check for the string that ends with ing such as going, seeing, ing, etc. |
. | Represent a single occurrance of any character except new line - a.b will check for the string that contains any character at the place of the dot such as acb, acbd, abbb, etc- .. will check if the string contains at least 2 characters |
*\* | Used to drop special meaning of a character following it or used to refer to a special character. - Since dot (.) is a metacharacter, so if you want to search it in a string you have to use the backslash (\) just before the dot (.) so that it will lose its specialty. |
[...] | Matches a single character in the listed set. If caret is the first character inside it, it means negation - [abc] means match any single character out of this set- [123] means match any single digit out of this set- [a-z] means match any single character out of lower case alphabets- [0-9] means match any single digit out of this set- [^0-3] means any number except 0, 1, 2, or 3- [^a-c] means any character except a, b, or c- [0-5][0-9] will match all the two-digits numbers from 00 to 59 - [0-9A-Fa-f] will match any hexadecimal digit.- Special characters lose their special meaning inside sets, so [(+*)] will match any of the literal characters '(', '+', '*', or ')'.- To match a literal ']' inside a set, precede it with a backslash, or place it at the beginning of the set, so [()[\]{}] and []()[{}] will both match parenthesis. |
^[...] | Matches any character in the set at the beginning of the string |
[^...] | Matches any character except those NOT in the listed set (negation) |
| | Or symbol works as the OR operator meaning it checks whether the pattern before or after the or symbol is present in the string or not - a|b will match any string that contains a or b such as acd, bcd, abcd, etc.- To match a literal '|', use | , or enclose it inside a character class, as in [|] . |
( ) | Used to capture and group |
Wild Card | Description |
---|---|
***** | The preceding character/expression is repeated zero or more times |
+ | The preceding character/expression is repeated one or more times, - ab+c will be matched for the string abc, abbc, dabc, but will not be matched for ac, abdc because there is no b in ac and d is not followed by c in abdc. |
? | The preceding character/expression is optional (zero or one occurrence). - ab?c will be matched for the string ac, abc, acb, dabc, dac but will not be matched for abbc because there are two b. Similarly, it will not be matched for abdc because b is not followed by c. |
{n,m} | The preceding character/expression is repeated from n to m times (both enclusive). - a{2,4} will be matched for the string aaab, baaaac, gaad, but will not be matched for strings like abc, bc because there is only one a or no a in both the cases. |
{n} | The preceding character/expression is repeated n times. - a{6} will match exactly six 'a' characters, but not five. |
{n,} | The preceding character/expression is repeated atleast n times |
{,m} | The preceding character/expression is repeated upto m times |
Note: The repeat characters (*
and +
) perform greedy search to match the largest possible string. However, you can performa a non greedy search by: *?
(0 or more characters but non-greedy) +?
(1 or more characters but non-greedy)
Code | Description |
---|---|
\d | Matches any decimal digit. This is equivalent to [0-9] |
\D | Matches any non-digit character. This is equivalent to [^0-9] or [^\d] |
\s | Matches any whitespace character. This is equivalent to [ \r\n\t\b\f] |
\S | Matches any non-whitespace character. This is equivalent to [^ \r\t\n\f] or [^\s] |
\w | Matches alphanumeric character. This is equivalent to [a-zA-Z0-9_] |
\W | Matches any non-alphanumeric character. This is equivalent to [^a-zA-Z0-9_] or [^\w] |
\b | Matches where the specified characters are at the beginning or at the end of a word r"\bain" OR r"ain\b" |
\B | Matches where the specified characters are present, but NOT or at the end of a word r"Bain" OR r"ain\B" |
re
Module¶import re
print(dir(re))
['A', 'ASCII', 'DEBUG', 'DOTALL', 'I', 'IGNORECASE', 'L', 'LOCALE', 'M', 'MULTILINE', 'Match', 'Pattern', 'RegexFlag', 'S', 'Scanner', 'T', 'TEMPLATE', 'U', 'UNICODE', 'VERBOSE', 'X', '_MAXCACHE', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_cache', '_compile', '_compile_repl', '_expand', '_locale', '_pickle', '_special_chars_map', '_subx', 'compile', 'copyreg', 'enum', 'error', 'escape', 'findall', 'finditer', 'fullmatch', 'functools', 'match', 'purge', 'search', 'split', 'sre_compile', 'sre_parse', 'sub', 'subn', 'template']
re.compile()
Method¶This method is used to compile a regular expression pattern into a regular expression object, which can be used for matching using its match()
, search()
and other methods.
re.compile(pattern, flags=0)
Where,
pattern
is the regular expression which you want to compile that you need to search/modify in a string or may be on a corpus of documents.flags
arguments can be used to modify the expression’s behaviour. Values can be any of the following variables, combined using bitwise OR (the | operator):IGNORECASE
or I
to do a case in-sensitive searchLOCALE
or L
to perform a locale aware match.MULTILINE
, M
to do multiline matching, affecting ^
and $
DOTALL
or S
to make the '.' special character match any character, including a newline; without this flag, '.' will match anything except a newline.Once you have an Pattern object
representing a compiled regular expression, you can use its methods to perform various operations on a string or may be in a corpus of documents:
p.match()
: Determine if the RE matches at the beginning of the string.p.search()
: Scan through a string, looking for any location where this RE matches.p.findall()
: Find all substrings where the RE matches, and returns them as a list.p.finditer()
: Find all substrings where the RE matches, and returns them as an iterator.p.split()
: Used to split string by the occurrences of pattern.p.sub()
: Used for for find and replace purpose..Note: Creating a regex object with re.compile
is highly recommended if you intend to apply the same expression to many strings; doing so will save CPU cycles.
import re
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)
print(p)
print(type(p))
re.compile('[A]+[a-z]*', re.MULTILINE) <class 're.Pattern'>
Once you have got a pattern object, you can use its various methods for searching from a string
re.Pattern.search()
Method¶p
produces a match, and return a corresponding match object. Return None if no position in the string matches the pattern.p.search(string, pos=0 endpos=9223372036854775807)
p
is the compiled pattern objectstring
is the test string from which we want to searchpos
and endpos
can be used to specify the portion of test string from where to searchimport re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)
match = p.search(str1)
print(match)
print(type(match))
<re.Match object; span=(6, 10), match='Arif'> <class 're.Match'>
re.Pattern.match()
Method¶p.match(string, pos=0 endpos=9223372036854775807)
p
is the compiled pattern objectstring
is the test string from which we want to searchpos
and endpos
can be used to specify the portion of test string from where to searchNote:
re.match()
will only match at the beginning of the string and not at the beginning of each line.search()
insteadimport re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)
rv = p.match(str1)
print(rv)
print(type(rv))
<re.Match object; span=(0, 4), match='Arif'> <class 're.Match'>
re.Pattern.findall()
Method¶search()
only returns the first match, match()
only matches at the beginning of the string, while findall()
returns all matches in a string.p
does not match, it returns an empty list.p.findall(string, pos=0 endpos=9223372036854775807)
p
is the compiled pattern objectstring
is the test string from which we want to searchpos
and endpos
can be used to specify the portion of test string from where to searchNote: The result depends on the number of capturing groups in the pattern. If there are no groups, return a list of strings matching the whole pattern. If there is exactly one group, return a list of strings matching that group. If multiple groups are present, return a list of tuples of strings matching the groups. Non-capturing groups do not affect the form of the result.
import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)
rv = p.findall(str1)
print(rv)
print(type(rv))
['Arif', 'Ahmad', 'AAA', 'As', 'Arif'] <class 'list'>
re.Pattern.finditer()
Method¶p.finditer(string, pos=0 endpos=9223372036854775807)
p
is the compiled pattern objectstring
is the test string from which we want to searchpos
and endpos
can be used to specify the portion of test string from where to searchimport re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
# The regular expression look for a string that starts with one or more uppercase 'A' alphabet, followed by
# zero or more lower case alphabets in multi-line mode
p = re.compile(r"[A]+[a-z]*", flags=re.M)
matches = p.finditer(str1)
print(matches)
print(type(matches))
<callable_iterator object at 0x7fd4d81359d0> <class 'callable_iterator'>
- Once we have got the iterator of
Match object
, we can iterate it using afor
loop.- Let us see how many match objects are there in this iterator named
matches
.
for m in matches:
print(m)
<re.Match object; span=(6, 10), match='Arif'> <re.Match object; span=(22, 27), match='Ahmad'> <re.Match object; span=(66, 69), match='AAA'> <re.Match object; span=(80, 82), match='As'> <re.Match object; span=(84, 88), match='Arif'>
- Every match object has many associated methods.
- Let us see different attributes of each match object using these methods.
The group()
method of the match object, return subgroups of the match (if they exist). By default return the entire match.
import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]*")
matches = p.finditer(str1)
for m in matches:
print(m.group(0))
Arif Ahmad AAA As Arif
The span()
method of the match object, return a 2-tuple containing the start and end index (end index not inclusive)
import re
str1 = "Rauf, Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]*")
matches = p.finditer(str1)
for m in matches:
print(m.span())
(6, 10) (22, 27) (66, 69) (80, 82) (84, 88)
The start(group=0)
method of the match object, return index of the start of the substring matched by group.
import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]+")
matches = p.finditer(str2)
for m in matches:
print(m.start())
4 20 78 82
The end(group=0)
method of the match object, return index of the end of the substring matched by group.
import re
str1 = "Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
str2 = "Mr. Arif, Jamil and Ahmad are good at playing acrobatic games. AAA is triple As. Arif Butt."
p = re.compile(r"[A]+[a-z]+")
matches = p.finditer(str2)
for m in matches:
print(m.end())
8 25 80 86
Read a text file
! cat datasets/names_addresses.txt
Mr. Arif Butt 615-555-7164 131 Model Town, Lahore 01-04-1975 arifpucit@gmail.com http://www.arifbutt.me Mrs. Nasira Jadoon 317.615.9124 33 Garden Town, Lahore 20/02/1969 nasira-123@gmail.com http://nasira.pu.edu.pk Mr. Khurram Shahzad 321#521#9254 69, A Wapda Town, Lahore 12.09.1985 khurram3@yahoo.com https://www.khurram.pu.edu.pk Ms Aqsa 123.555.1997 56 Joher Town, Lahore 12/08/2001 aqsa_007@gmail.com http://youtube.com Mr. B 321-555-4321 19 Township, Lahore 05-07-2002 mrB@yahoo.com http://facebook.com
with open("datasets/names_addresses.txt", "r") as fd:
print(fd.read())
Mr. Arif Butt 615-555-7164 131 Model Town, Lahore 01-04-1975 arifpucit@gmail.com http://www.arifbutt.me Mrs. Nasira Jadoon 317.615.9124 33 Garden Town, Lahore 20/02/1969 nasira-123@gmail.com http://nasira.pu.edu.pk Mr. Khurram Shahzad 321#521#9254 69, A Wapda Town, Lahore 12.09.1985 khurram3@yahoo.com https://www.khurram.pu.edu.pk Ms Aqsa 123.555.1997 56 Joher Town, Lahore 12/08/2001 aqsa_007@gmail.com http://youtube.com Mr. B 321-555-4321 19 Township, Lahore 05-07-2002 mrB@yahoo.com http://facebook.com
Let us read the data from the file in a string
with open("datasets/names_addresses.txt", "r") as fd:
teststring = fd.read()
teststring
'Mr. Arif Butt\n615-555-7164\n131 Model Town, Lahore\n01-04-1975\narifpucit@gmail.com\nhttp://www.arifbutt.me\n\n\nMrs. Nasira Jadoon\n317.615.9124\n33 Garden Town, Lahore\n20/02/1969\nnasira-123@gmail.com\nhttp://nasira.pu.edu.pk\n\n\n\nMr. Khurram Shahzad\n321#521#9254\n69, A Wapda Town, Lahore\n12.09.1985\nkhurram3@yahoo.com\nhttps://www.khurram.pu.edu.pk\n\n\nMs Aqsa\n123.555.1997\n56 Joher Town, Lahore\n12/08/2001\naqsa_007@gmail.com\nhttp://youtube.com\n\nMr. B\n321-555-4321\n19 Township, Lahore\n05-07-2002\nmrB@yahoo.com\nhttp://facebook.com'
import re
p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match)
p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
What if we want to get the complete name
p = re.compile(r'(Mr|Ms|Mrs)\.?\s\w+\s[A-Za-z]*')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
Mr. Arif Butt Mrs. Nasira Jadoon Mr. Khurram Shahzad Ms Aqsa Mr. B
p = re.compile(r'\d{2}.\d{2}.\d{4}')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
What if we just want to get the date, month and year separately
p = re.compile(r'(\d{2}).(\d{2}).(\d{4})')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
01-04-1975 20/02/1969 12.09.1985 12/08/2001 05-07-2002
Valid Name Part:
Valid Domain Part:
!cat datasets/names_addresses.txt
pattern = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-z]{2,3}')
#teststring contains data read from file
matches = pattern.finditer(teststring)
for match in matches:
print(match.group())
What if we want to just extract the usernames
p = re.compile(r'([a-zA-Z0-9_.+-]+)@([a-zA-Z0-9-]+)(\.[a-z]{2,3})')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
arifpucit@gmail.com nasira-123@gmail.com khurram3@yahoo.com aqsa_007@gmail.com mrB@yahoo.com
Assume that every valid phone number consiste of 10 digits in three groups of three, three and four digits. The three groups are separated by either a -
, .
or /
symbol
p = re.compile(r'\d{3}[./-]\d{3}[./-]\d{4}')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
You can easily extract the city codes, country codes and so on at your own by creating groups inside your regular expressions.
http://
or https://
www.
stringp = re.compile(r'https?://(www\.)?(\w+)(\.\w+)')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group())
Let us extract the top level domain (TLDs) only.
p = re.compile(r'https?://(www\.)?(\w+)(\.\w+)')
#teststring contains data read from file
matches = p.finditer(teststring)
for match in matches:
print(match.group(3))
Up to this point, we’ve simply performed searches against a static string. Regular expressions are also commonly used to modify strings in various ways, using the following pattern methods:
- split(): Split the string into a list, splitting it wherever the RE matches
- sub(): Find all substrings where the RE matches, and replace them with a different string
- subn(): Does the same thing as sub(), but returns the new string and the number of replacements
re.Pattern.split()
Method¶p.split(string, maxsplit=0)
string
: The variable pointing to the target string (i.e., the string we want to split).maxsplit
: The number of splits you wanted to perform. If maxsplit is 2, at most two splits occur, and the remainder of the string is returned as the final element of the list.Note:
It’s similar to the split()
method of strings but provides much more generality in the delimiters that you can split by; string split() only supports splitting by whitespace or by a fixed string.
# defining string
mystring = "My name is Arif Butt and my lucky number is 54 and 143"
mylist = mystring.split(sep=' ')
mylist
['My', 'name', '', '', '', '', '', '', '', '', '', '', '', '', 'is', 'Arif', 'Butt', 'and', 'my', 'lucky', 'number', 'is', '54', 'and', '143']
# importing required libraries
import re
# defining string
mystring = "My name is Arif Butt and my lucky number is 54 and 143"
p = re.compile(r"\s+")
word_list = p.split(mystring)
print(word_list)
['My', 'name', 'is', 'Arif', 'Butt', 'and', 'my', 'lucky', 'number', 'is', '54', 'and', '143']
The maxsplit
parameter of split()
is used to define how many splits you want to perform. In simple words, if the maxsplit is 2, then two splits will be done, and the remainder of the string is returned as the final element of the list.
import re
# defining string
mystring = "My name is Arif Butt and my lucky number is 54 and 143"
p = re.compile(r"\s+")
word_list = p.split(mystring, maxsplit=0)
print(word_list)
['My', 'name', 'is', 'Arif', 'Butt', 'and', 'my', 'lucky', 'number', 'is', '54', 'and', '143']
split()
method of strings allows you to split by whitespace or by a fixed string.split()
method allows you to specify a regex pattern for the delimiters where you can specify multiple delimiters.comma
or by space
.comma
or by hyphen
.import re
# defining string
mystring = "12,45,78,85-17-89"
p = re.compile(r"-|,")
p = re.compile(r"[-,]")
word_list = p.split(mystring)
print(word_list)
['12', '45', '78', '85', '17', '89']
import re
# defining string
mystring = "12and45, 78and85-17and89-97,54"
p = re.compile(r"and|[\s,-]+")
word_list = p.split(mystring)
print(word_list)
['12', '45', '78', '85', '17', '89', '97', '54']
re.Pattern.sub()
and re.Pattern.subn()
Methods¶sub()
the subn()
methods to search
and replace
patterns in a string. Using these methods we can replace one or more occurrences of a regex pattern in the target string with a substitute string.The sub()
method return the string obtained by replacing the leftmost non-overlapping occurrences of pattern in string
by the replacement repl
.
p.sub(repl, string, count=0)
Where,
repl
: The replacement that we are going to insert for each occurrence of a pattern. The replacement can be a string or function.string
: The variable pointing to the target string (In which we want to perform the replacement).count
: The default value of count is zero, means, find and replace all occurrences of pattern with replacement. For count=n, means replace first n occurrencesof pattern with the replacementIt returns the string obtained by replacing the pattern occurrences in the string with the replacement string. If the pattern isn’t found, the string is returned unchanged.
Replace white space with underscore character
import re
# defining string
mystring = "Learning is fun with Arif Butt"
p = re.compile(r"\s")
word_list = p.sub("_", mystring)
print(word_list)
Learning_is_fun_with_Arif_Butt
Remove whitespaces from a string
import re
# defining string
mystring = "Learning is fun with Arif Butt"
p = re.compile(r"\s+")
word_list = p.sub("", mystring)
print(word_list)
LearningisfunwithArifButt
Remove leading Spaces from a string
import re
# defining string
mystring = " Learning is fun with Arif Butt"
# ^\s+ remove only leading spaces
# caret (^) matches only at the start of the string
p = re.compile(r"^\s+")
word_list = p.sub("", mystring)
print(word_list)
Learning is fun with Arif Butt
Remove both leading and trailing spaces
import re
# defining string
mystring = " Learning is fun with Arif Butt \t. "
# ^\s+ remove leading spaces
# ^\s+$ removes trailing spaces
p = re.compile(r"^\s+|\s+$")
word_list = p.sub("", mystring)
print(word_list)
Learning is fun with Arif Butt .
string1 = 'a aa ab bc bb abc abcd ba aaaa'
p = re.compile("[abc]")
matches = p.finditer(string1)
for m in matches:
print(m)
<re.Match object; span=(0, 1), match='a'> <re.Match object; span=(2, 3), match='a'> <re.Match object; span=(3, 4), match='a'> <re.Match object; span=(5, 6), match='a'> <re.Match object; span=(6, 7), match='b'> <re.Match object; span=(8, 9), match='b'> <re.Match object; span=(9, 10), match='c'> <re.Match object; span=(11, 12), match='b'> <re.Match object; span=(12, 13), match='b'> <re.Match object; span=(14, 15), match='a'> <re.Match object; span=(15, 16), match='b'> <re.Match object; span=(16, 17), match='c'> <re.Match object; span=(18, 19), match='a'> <re.Match object; span=(19, 20), match='b'> <re.Match object; span=(20, 21), match='c'> <re.Match object; span=(23, 24), match='b'> <re.Match object; span=(24, 25), match='a'> <re.Match object; span=(26, 27), match='a'> <re.Match object; span=(27, 28), match='a'> <re.Match object; span=(28, 29), match='a'> <re.Match object; span=(29, 30), match='a'>