Our goal is to build the vocabulary, so the first step is to get all the words from the corpus.

Task: Get All the words from the corpus

Run the code to get the required imports:

from nltk import word_tokenize

words = []

Iterate through all the documents, use word_tokenize in every one of them and append tokens to words. Remember to bring the words to the lowercase value (use lower() function).

Once you have all the words use FreqDist class to get the frequencies. We've written a function for you to get the top words from the vocabulary you will be building.

from nltk.probability import FreqDist

def get_vocabulary(words, top=500):
    word_counts = FreqDist(words)
    vocabulary = word_counts.most_common(top)
    return vocabulary

get_vocabulary(words)

As we want our vocabulary to contain only "real" words, you should get rid of the words that are numbers, punctuations, etc. One way to do it is to use isalpha function, another is to use regular expressions.

Task

Use your preferred way to remove non-words from our vocabulary (words variable).

Once done this line should return the vocabulary with only alphabetical characters.

get_vocabulary(words)

The vocabulary looks better now, but there are still some issues. Here we will deal with stop words which are commonly appearing words that don't bring much meaning of context. These are words like:

a, an, and, both, do, have, how, is, it, I, more, much, my, on, one, so, the, this, to, too, very, what, who, where, you

... and many more. Every language have their on list and different algorithms and teams use slightly changed lists. Sometimes you may need to decide the list of your stop words. We will use nltk.corpus.stopwords for english language.

from nltk.corpus import stopwords
stop_words = stopwords.words('english')

Task

Filter stop words from the words variable.

Once done this line should return the vocabulary without stop words.

get_vocabulary(words)

In some situations it's good to use stemming with the vocabulary. In a nutshell it is bringing the word to it's root value. You may have notices that we have for example sets like: acquire - acquired, firm - firms or product - production being set as separate features. We may want to merge to the base values.

There are several stemmers out there so you don't have to write your own (it's actually quite a complex although interesting task). We will use PorterStemmer imported from nltk library. This time our vectorizer construction has to change a little bit. We will first build the analyzer and then use it to initialize the final one.

from nltk.stem.porter import PorterStemmer
stemmer = PorterStemmer()

Task

Replace every word with the stemmed value (using stem function on the stemmer).

Once done this line should return the vocabulary of stems.

get_vocabulary(words)

If you agree on the vocabulary to be ready, the last step is to use it to build a vector of features for the documents in the corpus. This would mean going through every document and comparing the words with the vocabulary while getting the words frequencies for specific document.

We've built a function for you to do it for specific document.

import numpy as np

def vectorize_doc(document, vocabulary):
    doc_words = word_tokenize(document.lower())
    doc_words = [stemmer.stem(word) for word in doc_words]
    doc_frequencies = FreqDist(doc_words)
    voc_size = len(vocabulary)
    doc_vector = np.zeros(voc_size)
    for idx in range(0, voc_size):
        token = vocabulary[idx]
        if token in doc_frequencies.keys():
            doc_vector[idx] = doc_frequencies[token]
    return doc_vector

Once you have this function see how the vector looks like for different documents.

vocabulary = [w for w, _ in get_vocabulary(words)]

idx = 1467
vectorize_doc(documents[idx], vocabulary)

Change the index and see other documents.