Let's take a look at a sample movie review from a movie data corpus that is included in nlkt:

import data_reader sample_movie_review = data_reader.read_reviews()[123] print(sample_movie_review)

In NLTK, the default sentence tokenizer is called the Punkt tokenizer. Let's load the Punkt tokenizer:

import nltk.data sentence_tokenizer = nltk.data.load('tokenizers/punkt/PY3/english.pickle')

Let's create a sentences array and observe how well the punkt tokenizer was able to split the movie review into sentences:

sentences = sentence_tokenizer.tokenize(sample_movie_review) print('\n------\n'.join(sentences))

In order to do word tokenization we have a few choices of tokenizer. Let's set a sentence we will try to tokenize:

sent = "NATO countries did not commit to Trump's call for 4% spending, sources say"

We can use the Penn TreeBank word tokenizer which uses regular expressions to tokenize text:

from nltk.tokenize import TreebankWordTokenizer treebank_word_tokenizer = TreebankWordTokenizer() treebank_word_tokenizer.tokenize(sent)

or we can use the WordPuncTokenizer that splits all punctuations into separate tokens:

from nltk.tokenize import WordPunctTokenizer word_punct_tokenizer = WordPunctTokenizer() word_punct_tokenizer.tokenize(sent)

There is also a tokenizer specialized to parsing Tweets. Let's pick a sample tweet:

tweet = "@remy This is waaaaayyyy too much for you!!!!!! #funnygetups :-P"

and let's try to run the tweet through the tweet tokenizer from nltk.tokenize import TweetTokenizer tweet_tokenizer = TweetTokenizer(strip_handles=True) tweet_tokenizer.tokenize(tweet)

Let's create a text string we will use for determining language

ENGLISH: text="Just once I'd like someone to call me 'sir' without adding 'you're making a scene.'" FRENCH: text="Après avoir rencontré Theresa May, le président américain devait rejoindre le palais de Windsor pour prendre le thé avec la reine Elizabeth II." RUSSIAN: text="Более 50 тысяч старых квартир выставили на продажу в Подмосковье"

Now, let's tokenize the text using the WordPunct tokenizer because we would like the punctuation marks to be exluded from the words:

from nltk import wordpunct_tokenize wordpunct_tokenize(text)

In order to build our language analyzer we will look at stop words that are present in different languages but let's see which languages nltk supports right from the box:

from nltk.corpus import stopwords stopwords.fileids()

Let's take a closer look at the words that are present in the English language:

stopwords.words('english')[0:10]

Using the stopwords let's build a simple language identifier that will count how many words in our sentence appear in a particular language's stop word list as stop words are very common generally:

from nltk import wordpunct_tokenize from nltk.corpus import stopwords languages_ratios = {} tokens = wordpunct_tokenize(text) words = [word.lower() for word in tokens] for language in stopwords.fileids(): stopwords_set = set(stopwords.words(language)) words_set = set(words) common_elements = words_set.intersection(stopwords_set) languages_ratios[language] = len(common_elements) # language "score"

Let's see how the language detector scored our sentence:

languages_ratios

And for completeness let's just select the language with the biggest score:

most_applicable_language = max(languages_ratios, key=languages_ratios.get) most_applicable_language

Let's pick a text with an acronym:

text="Pep Guardiola, NBA & NFL helped turn England into potential world champions"

Simple dictionary:

abbrevs={'NBA':'National Basketball Association','NFL':'National Football League'} for abbrev in abbrevs: text= text.replace(abbrev,abbrevs[abbrev])

Let's see how it was normalized: print(text)

NLTK implements a number stemmers. The popular "Porter Stemmer" which we will examine below as well as other stemmers including the "Lancaster Stemmer" and the "Snowball Stemmer".

Let's take a look at the Porter Stemmer:

from nltk.stem.porter import PorterStemmer porter_stemmer = PorterStemmer()

Let's see how the porter stemmer would stem:

porter_stemmer.stem("run") porter_stemmer.stem("ran") porter_stemmer.stem("running") porter_stemmer.stem("geese") porter_stemmer.stem("goose")

NLTK also implements a Lemmatizer based on the WordNet searchable database:

from nltk.stem import WordNetLemmatizer wordnet_lemmatizer = WordNetLemmatizer()

Let's see how the lemmatizer deals with the same cases:

from nltk.corpus import wordnet wordnet_lemmatizer.lemmatize("run", wordnet.VERB) wordnet_lemmatizer.lemmatize("ran", wordnet.VERB) wordnet_lemmatizer.lemmatize("running", wordnet.VERB) wordnet_lemmatizer.lemmatize("geese") wordnet_lemmatizer.lemmatize("goose")

NLTK has a trained part of speech model that can be used:

sent = "Geoff Boeing, a postdoc in the Urban Analytics Lab at the University California, Berkeley, has published a blog post that offers a fascinating look at the street orientation of major cities in the USA and around the world."

We can invoke the Parts-Of-Speech tagger as follows:

import nltk word_list = nltk.word_tokenize(sent) pos_tagged_tree = nltk.pos_tag(word_list) print(pos_tagged_tree)

To discover what the tags mean you can request all the tags that are used by the UPenn TreeBank:

nltk.help.upenn_tagset()

out-of-the-box, NLTK provides a classifier that has been trained to recognize named entities.

Let's take a look at the following sentence: sentence = "Anderson says the Baillie Gifford firm still supports Elon Musk's efforts in the United States to promote clean energy but would like peace and execution at this stage"

Let's see if NLTK's pre-trained model will be able to decipher the entities in this sentence: import nltk word_list = nltk.word_tokenize(sentence ) pos_tagged_tree = nltk.pos_tag(word_list) entity_tree = nltk.ne_chunk(pos_tagged_tree, binary=False) print(entity_tree)