Different functions to tokenize text.
from mlxtend.text import tokenizer_[type]
Different functions to tokenize text for natural language processing tasks, for example such as building a bag-of-words model for text classification.
from mlxtend.text import tokenizer_emoticons
tokenizer_emoticons('</a>This :) is :( a test :-)!')
[':)', ':(', ':-)']
from mlxtend.text import tokenizer_words_and_emoticons
tokenizer_words_and_emoticons('</a>This :) is :( a test :-)!')
['this', 'is', 'a', 'test', ':)', ':(', ':-)']
with open('../../api_modules/mlxtend.text/tokenizer_emoticons.md', 'r') as f:
s = f.read() + '<br><br>'
with open('../../api_modules/mlxtend.text/tokenizer_words_and_emoticons.md', 'r') as f:
s2 = f.readlines()
s += ''.join(s2[1:])
print(s)
## tokenizer_emoticons *tokenizer_emoticons(text)* Return emoticons from text **Examples** >>> tokenizer_emoticons('</a>This :) is :( a test :-)!') [':)', ':(', ':-)'] For usage examples, please see [http://rasbt.github.io/mlxtend/user_guide/text/tokenizer_emoticons/](http://rasbt.github.io/mlxtend/user_guide/text/tokenizer_emoticons/) <br><br> *tokenizer_words_and_emoticons(text)* Convert text to lowercase words and emoticons. **Examples** >>> tokenizer_words_and_emoticons('</a>This :) is :( a test :-)!') ['this', 'is', 'a', 'test', ':)', ':(', ':-)'] For more usage examples, please see [http://rasbt.github.io/mlxtend/user_guide/text/tokenizer_words_and_emoticons/](http://rasbt.github.io/mlxtend/user_guide/text/tokenizer_words_and_emoticons/)