Regular expressions and word tokenization¶

This chapter will introduce some basic NLP concepts, such as word tokenization and regular expressions to help parse text. You'll also learn how to handle non-English text and more difficult tokenization you might find. This is the Summary of lecture "Introduction to Natural Language Processing in Python", via datacamp.

• toc: true
• badges: true
• comments: true
• author: Chanseok Kang
• categories: [Python, Datacamp, Natural_Language_Processing]
• image: images/line_num_words.png

Introduction to regular expressions¶

• Regular expressions
  • Strings with a special syntax
  • Allow us to match patterns in other strings
  • Applications of regular expressions
    • Find all web links in a document
    • Parse email addresses, remove/replace unwanted characters
• Common Regex patterns

• Python's re Module
  • split: split a string on regex
  • findall: find all patterns in a string
  • search: search for a pattern
  • match: match an entire string or substring based on a pattern
  • Pattern first, and the string second
  • May return an iterator, string, or match object

Practicing regular expressions - re.split() and re.findall()¶

Now you'll get a chance to write some regular expressions to match digits, strings and non-alphanumeric characters. Take a look at my_string first by printing it in the IPython Shell, to determine how you might best match the different steps.

Note: It's important to prefix your regex patterns with r to ensure that your patterns are interpreted in the way you want them to. Else, you may encounter problems to do with escape sequences in strings. For example, "\n" in Python is used to indicate a new line, but if you use the r prefix, it will be interpreted as the raw string "\n" - that is, the character "\" followed by the character "n" - and not as a new line.

Remember from the video that the syntax for the regex library is to always to pass the pattern first, and then the string second.

Introduction to tokenization¶

• Tokenization
  • Turning a string or document into tokens (smaller chunks)
  • One step in preparing a text for NLP
  • Many different theories and rules
  • You can create your own rules using regular expressions
  • Some examples:
    • Breaking out words or sentences
    • Separating punctuation
    • Separating all hashtags in a tweet
• Why tokenize?
  • Easier to map part of speech
  • Matching common words
  • Removing unwanted tokens
• Other nltk tokenizers
  • sent_tokenize: tokenize a document into sentences
  • regexp_tokenize: tokenize a string or document based on a regular expression pattern
  • TweetTokenizer: special class just for tweet tokenization, allowing you to separate hashtags, mentions and lots of exclamation points

Word tokenization with NLTK¶

Here, you'll be using the first scene of Monty Python's Holy Grail, which has been pre-loaded as scene_one.

Your job in this exercise is to utilize word_tokenize and sent_tokenize from nltk.tokenize to tokenize both words and sentences from Python strings - in this case, the first scene of Monty Python's Holy Grail.

Note: Before using NLTK, you must install punkt package for tokenizer

More regex with re.search()¶

In this exercise, you'll utilize re.search() and re.match() to find specific tokens. Both search and match expect regex patterns, similar to those you defined in an earlier exercise. You'll apply these regex library methods to the same Monty Python text from the nltk corpora.

Advanced tokenization with NLTK and regex¶

• Regex groups using or |
  • OR is represented using |
  • You can define a group using ()
  • You can define explicit character ranges using []
• Regex ranges and groups

Choosing a tokenizer¶

Given the following string, which of the below patterns is the best tokenizer? If possible, you want to retain sentence punctuation as separate tokens, but have '#1' remain a single token.

my_string = "SOLDIER #1: Found them? In Mercea? The coconut's tropical!"

Additionally, regexp_tokenize has been imported from nltk.tokenize. You can use regexp_tokenize(string, pattern) with my_string and one of the patterns as arguments to experiment for yourself and see which is the best tokenizer.

Regex with NLTK tokenization¶

Twitter is a frequently used source for NLP text and tasks. In this exercise, you'll build a more complex tokenizer for tweets with hashtags and mentions using nltk and regex. The nltk.tokenize.TweetTokenizer class gives you some extra methods and attributes for parsing tweets.

Here, you're given some example tweets to parse using both TweetTokenizer and regexp_tokenize from the nltk.tokenize module.

Unlike the syntax for the regex library, with nltk_tokenize() you pass the pattern as the second argument.

Non-ascii tokenization¶

In this exercise, you'll practice advanced tokenization by tokenizing some non-ascii based text. You'll be using German with emoji!

Here, you have access to a string called german_text, which has been printed for you in the Shell. Notice the emoji and the German characters!

Unicode ranges for emoji are:

('\U0001F300'-'\U0001F5FF'), ('\U0001F600-\U0001F64F'), ('\U0001F680-\U0001F6FF'), and ('\u2600'-\u26FF-\u2700-\u27BF').

Charting word length with NLTK¶

Charting practice¶

Try using your new skills to find and chart the number of words per line in the script using matplotlib. The Holy Grail script is loaded for you, and you need to use regex to find the words per line.

Using list comprehensions here will speed up your computations. For example: my_lines = [tokenize(l) for l in lines] will call a function tokenize on each line in the list lines. The new transformed list will be saved in the my_lines variable.