Our task is to build a classifying neural network with TensorFlow. First, we need set up the architecture, train the network (using training set) and then evaluate the result on the test set.

The following image shows the classification process and the layers of the neural network:

To feed the network with the training data, we need to flatten the digit images. Depending on the phase (training or testing), different examples will be pushed through the classifier. The training process will be based on the labels while comparing them to the current predictions. This is why we need to define these two placeholders.

# Define placeholders
training_data = tf.placeholder(tf.float32, [None, image_size*image_size])
labels = tf.placeholder(tf.float32, [None, labels_size])

Placeholders will be filled with the values passed when evaluating the computation graph. But the actual goal of the training is to adjust the values of the weights and biases. This is why we need the structure that will allow changing the values along the process.

TensorFlow provides variables for this exact purpose. The initial values for the weights will follow the normal distribution while biases will get the value 1.0. Once we have them defined, the creation of the output layer is just one line.

# Variables to be tuned
W = tf.Variable(tf.truncated_normal([image_size*image_size, labels_size], stddev=0.1))
b = tf.Variable(tf.constant(0.1, shape=[labels_size]))

# Build the network (only output layer)
output = tf.matmul(training_data, W) + b

Training process works by optimising (either maximising or minimising) the loss function. In our case, we would like to minimise the difference between the network predictions and actual labels' values. In deep learning, we often use a technique called Cross entropy to define the loss.

TensorFlow provides the function called tf.nn.softmax_cross_entropy_with_logits that internally applies the softmax on the model's unnormalized prediction and sums across all classes. The tf.reduce_mean function takes the average over these sums. This way we get the function that can be further optimised. In our example, we use gradient descent method from the tf.train API.

# Define the loss function
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=output))

# Training step
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

Gradient descent optimiser will work in several steps adjusting the values of the W and b variables. We would also like to have a way of evaluating the performance.

First, we want to compare which labels were predicted correctly by using tf.argmax function. tf.equal returns the list of booleans so by casting the values to float and then calculating the average we finally get the accuracy of the model.

# Accuracy calculation
correct_prediction = tf.equal(tf.argmax(output, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Now is the time when we have the computational graph built, and we can start the training process. To start, we need to initialise the session and variables defined earlier.

# Run the training
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

As mentioned before, optimiser works in steps. In our case, we run the train_step inside the loop feeding it with the batch data: images and corresponding labels. This is the moment where we fill in the placeholders by using feed_dict parameters of the function run.

for i in range(steps_number):
  # Get the next batch
  input_batch, labels_batch = mnist.train.next_batch(batch_size)
  feed_dict = {training_data: input_batch, labels: labels_batch}

  # Run the training step
  train_step.run(feed_dict=feed_dict)

We can make use of the accuracy defined previously to monitor the performance on the batches during the training process. By adding the following code, we will print out the value every 100 steps.

  # Print the accuracy progress on the batch every 100 steps
  if i%100 == 0:
    train_accuracy = accuracy.eval(feed_dict=feed_dict)
    print("Step %d, training batch accuracy %g %%"%(i, train_accuracy*100))

After the training is finished we would like to check the network performance on the data it hasn't previously seen - in the test set. We can reuse accuracy and feed it with the training data instead of the training batch.

# Evaluate on the test set
test_accuracy = accuracy.eval(feed_dict={training_data: mnist.test.images, labels: mnist.test.labels})
print("Test accuracy: %g %%"%(test_accuracy*100))

The file is ready now, and we can run it using the following command:

python app.py

Try to change some values like learning_rate or steps_number to see if you can influence the accuracy.