Let's first download and load the data. Let us define the source:

dataset_uri = "http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz" dataset_name = "aclImdb"

The data will be stored in the train_data and test_data variables.

dataset_path = help.download_dataset(dataset_name, dataset_uri, True) train_data, test_data = help.load_data(dataset_path, dataset_name)

You can have a look a the headers by running:

train_data.head()

And:

test_data.head()

The classificator will be trained to fit the sentiment feature.

For the sentece to be the input of the neural network it has to be transformed into a set of features. In this example we are going to use nnlm_en_dim128_encoder, the encoder available from TensorFlow Hub that is performing this transformation. The following code is preparing the column from the free text:

text_column = tf_hub.text_embedding_column("review", "https://tfhub.dev/google/nnlm-en-dim128/1")

Once we have the features defined we can build canned estimaton that will prepare the fully connected neural network. The code below is building the network with 2 hidden layers (of sizez 512 and 128 respectively). It uses Adam Optimizer with the learning rate of 0.003.

sentiment_classifier = tf.estimator.DNNClassifier(
    hidden_units=[512, 128],
    feature_columns=[text_column],
    n_classes=2,
    optimizer=tf.train.AdagradOptimizer(learning_rate=0.003))

Once the network is set up it has to be trained. To do this you have to define the input function:

input_fn = tf.estimator.inputs.pandas_input_fn(train_data, train_data['sentiment'], num_epochs=None, shuffle=True)

Now we will use it to train the model for 1000 steps:

sentiment_classifier.train(input_fn=input_fn, steps=1000)

To see the result of the training we will use evaluate function, but first let use define the reusable function to be fed as the input function to evaluate and predict methods:

def predict_input_fn(data, target): return tf.estimator.inputs.pandas_input_fn(data, data[target], shuffle=False)

Let's see how the model performs for the training and testing data:

train_eval = sentiment_classifier.evaluate(input_fn=predict_input_fn(train_data, 'sentiment'))
test_eval = sentiment_classifier.evaluate(input_fn=predict_input_fn(test_data, 'sentiment'))

print("Training accuracy: {accuracy}".format(**train_eval))
print("Test accuracy: {accuracy}".format(**test_eval))

Once you're confident about the model performance you can use it for predictions. Let us define a function that will get new reviews and print the prediction results:

def predict_sentiment(reviews): predict_input_fn = tf.estimator.inputs.pandas_input_fn(pd.DataFrame({'review': reviews}), shuffle=False) predictions = sentiment_classifier.predict(input_fn=predict_input_fn) for index, prediction in enumerate(predictions): print("Review: {} --> Sentiment: {}".format(new_reviews[index], "Negative" if prediction["classes"][0] == b'0' else "Positive"))

We can define some new reviews: new_reviews = ["This was an awful movie", "I think it was great"]

And then check about the train model predictions:

predict_sentiment(new_reviews)