This material was developed to help those interested in the physics of the D-Wave One^TM System to perform some simple experiments and explore some of the interesting science behind the processor technology. This tutorial series is designed to introduce you to quantum computer programming using a practical, hands-on approach. By the end of this tutorial you should understand the way that the processor operates on a device level, and have learned how to create, manipulate and measure the properties of spin chains.

All of the software necessary for developers to begin learning how to program D-Wave quantum computing systems can be downloaded from the DOWNLOAD DEVKITS page. As the developer portal is currently in BETA test stage, you will need to have a D-Wave developer account to download these packs. To follow this tutorial it will be necessary to familiarize yourself with the BlackBox Compiler tutorial and the Quantum Unsupervised Feature Learning tutorial.

The high level programming language used in the included examples and source code is Python 2.6. Some experience of using the Python programming language is helpful but not required.

What you will learn

By following through the material in this white paper, you will learn:

What a neural net is and why they are useful for artificial intelligence / machine learning applications

How to write a program that constructs, randomizes, activates, and trains a neural network

How to feed image data through the neural net in a pre-compressed form

How to use the quantum hardware through the BlackBox compiler to accelerate the training of such a network

Software requirements

In order to proceed you should install several free software packages.

Python - The material here assumes you are using Python 2.6. You can download Python from http://www.Python.org. If you need a programming development environment, Pycharm from Jetbrains http://www.jetbrains.com/pycharm/ is excellent. There is a small license fee to use it.

NumPy - This is a scientific computing library. It is at http://numpy.scipy.org/

Python Imaging Library (PIL) - This is a free graphics library. It is at http://www.pythonware.com/products/pil/

D-Wave Python developer pack - The D-Wave Python developer pack can be downloaded from the DOWNLOAD DEVKITS page

Code for this tutorial (optional - i.e. if you want to see the full code files for each section). You can find the code referenced in this tutorial here: Github Repository for neural-net code. You will need a github account to access this code. Once you have a github account you can email devportal at dwavesys dot com and we can add you to the github D-Wave repositories.

You will need to download the training image data files separately from the source code. They can be found here: Images for QUFL .zip file. Warning: this is a large (22MB) file.

Accompanying Python file naming conventions

The structuring of this tutorial is such that code developed in each section is labelled as such in the repository, etc. So for section 1 the main routine is neural-net-tutorial1.py. Code from each section should all work independently. Unfortunately writing the code in this way means that everything has had to be kept in a single Python file for each section, including all function and class definitions. This is certainly not the elegant Python way, but it avoids people getting confused about which versions of files are required for each section, and hopefully it will be useful for people to be able to see how the code is built up from section to section. It also avoids lengthy descriptions of where to put each code snippet which would change with every section if multiple files were used.

Other files needed that are not in the main routine built up in this tutorial are: compressing_an_image.py, DX.txt, (these are all available through the github link in software requirements above) and the folders of training images used for this tutorial (again follow the link in software requirements above). Warning: this is a large (22MB) file. Unzip and place this images folder in your directory with the source code. You may already have this data downloaded if you followed through the QUFL tutorial.

SECTION 1

1.1 - Introduction

The idea of this tutorial is to explore training a multi-layer neural net using the hardware without the need for layer-by-layer training which makes many NN implementations intractable. The purpose of the net is to classify incoming images.

1.2 - Neural net structure

In this tutorial we'll consider a neural net with 3 layers, an input layer, an output layer and a hidden layer, like this:

Figure 1. An example of a 3-layer neural net

The black spot are known as neurons, or nodes, and the lines are known as synapses, or connections. Here the net is shown fully connected. This is the set of all possible synapses. Note the structure - nodes can only connect to nodes in neighbouring layers ' and no connections within layers are allowed.

The user is allowed to set the number of nodes in each layer, but as we work through the code it will become clear why having the number of nodes set to 8-6-4 makes sense for this application.

So what do we mean by 'training' such a neural net and why would we want to do that? Well, training involves selecting a subset of synaptic connections so that our neural network embodies some useful behaviour. So the first important thing to note is that we are trying to select an optimal subset of synaptic connections. We use a binary variable (0 or 1) to denote the presence or absence of a synaptic connection. Using this notation, in the figure above all the connections would be set to 1. By restricting the network to only have connections specified by binary variables, we start to see how this is shaping up to be a binary optimization problem, and optimization problems are of course what the D-Wave One processor is designed to solve efficiently. So the variables over which you wish to optimize are potential connections between nodes.

1.3 - Creating neurons and synapses

First we will set up our Python file by importing the Python Imaging library, PIL, which can be found here. This will allow us to make a simple visualization of our network and save it to disk as a bitmap file. We also need to import the built-in module random, as we are going to be randomizing the connections in the network very shortly.

Code Snippet : A couple of imports

from PIL import Image, ImageDraw
import random

The next thing we would like to do is to create some structures to represent the neurons (nodes) and synapses (connections) in the network. Because we'll be creating many such structures we'll make classes for neurons and synapses. What would we want to do with a neuron and a synapse? Well, in this tutorial we'll want to be able to query each object to know if it is inactive or active, and we also want to be able to change its status from time to time. We therefore want 3 methods for each class, one that gives us an object's number, one that gets its status, and one that sets its status. This is implemented as follows:

Code Snippet : Create class structures to describe neurons and synapses

        class Neuron:
    '''A neuron is a node in the network.'''
    def __init__(self, node_num, status, activation_input):
        self.node_num = node_num
        self.status = status
        self.activation_input = activation_input

    def get_node_num(self):
        '''Returns the number of the node in a layer.'''
        return self.node_num

    def get_status(self):
        '''Returns the activation status of a node - either 0 or 1.'''
        return self.status

    def set_status(self, status):
        '''Sets a node active(1) or inactive (0)'''
        if status == 1 or status == 0:
            self.status = int(status)
        else:
            print 'Invalid status setting: Should be 0 or 1 only'

class Synapse:
    '''A synapse is a connection between two nodes.'''
    def __init__(self, conn_num, status):
        self.conn_num = conn_num
        self.status = status

    def get_conn_num(self):
        '''Returns the number of the synapse in a layer.'''
        return self.conn_num

    def get_status(self):
        '''Returns the status of a connection - either 0 or 1.'''
        return self.status

    def set_status(self, status):
        '''Sets the status of a connection - either 0 or 1.'''
        if status == 1 or status == 0:
            self.status = int(status)
        else:
            print 'Invalid status setting: Should be 0 or 1 only'

        

1.4 - Constructing the net

Now that we have neuron and synapse classes, we can start building a net! We want the user to be able to decide the number of nodes per layer, so we'll define variables to hold those values.

Code Snippet : Function to build the net and define parameters

        def load_net_parameters():

##############################################################################################
######################         CREATE THE NET              ###################################

# create the neural net - all nodes in all layers stored in a single list.
#Note that numbering of the nodes is done uniquely. Here is an example of how a net is numbered:

#      16 17 18 19
#   9  10 11 12 13 14
#  1  2  3  4  5  6  7  8
##############################################################################################

    number_nodes_layer_1 = 8
    number_nodes_layer_2 = 6
    number_nodes_layer_3 = 4

    threshold_activation = 0

    dim_x_viz = 23 #For the visualization, sets the size of the little images
    dim_y_viz = 33
    dim_x_viz_big = 80 # The larger versions of the same dictionary atoms
    dim_y_viz_big = 112

    nodes = []
    conns = dict()

    total_number_nodes = number_nodes_layer_1+number_nodes_layer_2+number_nodes_layer_3
    total_number_conns = number_nodes_layer_1*number_nodes_layer_2+number_nodes_layer_2*number_nodes_layer_3

    layer1_index = range(number_nodes_layer_1)
    layer2_index = range(number_nodes_layer_1, number_nodes_layer_1+number_nodes_layer_2)
    layer3_index = range(number_nodes_layer_1+number_nodes_layer_2, number_nodes_layer_1 \
                         + number_nodes_layer_2+number_nodes_layer_3)

    for i in range(total_number_nodes):
        nodes.append(Neuron(i, 0, 0))

    nodes_layer1 = []
    nodes_layer2 = []
    nodes_layer3 = []

    for each in layer1_index:
        nodes_layer1.append(nodes[each])

    for each in layer2_index:
        nodes_layer2.append(nodes[each])

    for each in layer3_index:
        nodes_layer3.append(nodes[each])

    #create the synaptic connection list, populates a dictionary to hold the synaptic objects

    for i in layer1_index: #make connections between layer 1 and layer 2 nodes
        for j in layer2_index:
            conns[(i,j)]=Synapse((i,j), 1)

    for i in layer2_index: #make connections between layer 2 and layer 3 nodes
        for j in layer3_index:
            conns[(i,j)]=Synapse((i,j), 1)


    return nodes_layer1, nodes_layer2, nodes_layer3, number_nodes_layer_1, number_nodes_layer_2, \
            number_nodes_layer_3, threshold_activation, dim_x_viz, dim_y_viz, dim_x_viz_big, \
            dim_y_viz_big, nodes, conns, total_number_nodes, total_number_conns, \
            layer1_index, layer2_index, layer3_index

1.5 - Randomizing the net

So far all our synaptic connections have been initialized to 1, giving a fully connected net as shown in figure 1. Any neural net will work by adjusting its connections based on some user input. to begin exploring changing the connection status, let's try randomizing the connections in the network. We write a simple function to set the status of each connection to a randomly chosen 0 or 1.

Code Snippet : Function to randomize the net

def randomize_net(conns):
    randomized_connections = []
    for i, j in conns.iteritems():
        if random.choice([0,1]):
            j.set_status(1)
            randomized_connections.append(1)
        else:
            j.set_status(0) #Set to 0 for randomly connected net, 1 for a fully connected net
            randomized_connections.append(0)

1.6 - Net visualization

Before we go any further, we'll create the network structure (shown above) in code, and write a visualization routine. The visualization will help us understand what is happening and debug any potential problems, as it can be hard to understand what the nodes and connections are doing without a graphical representation of the net.

The visualization code is mainly a debugging and teaching aid, so we will make sure that it can be turned on and off when we run the main routine (we don't want to be visualizing the net whilst training it or else the code will run very slowly indeed).

Code Snippet : Function to visualize the net

def visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns):
    bitmap_dimension = (600,400)
    im = Image.new('RGB', bitmap_dimension, 'white')
    draw = ImageDraw.Draw(im)
    circlesize = 6

    node_coords = [] # the co-ordinate list will have the same indexing as the node ID

    for j in range(len(nodes_layer1)):
        if nodes_layer1[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 100
        y = j*bitmap_dimension[1]/(len(nodes_layer1))+0.5*(bitmap_dimension[1]/(len(nodes_layer1)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    for j in range(len(nodes_layer2)):
        if nodes_layer2[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 300
        y = j*bitmap_dimension[1]/(len(nodes_layer2))+0.5*(bitmap_dimension[1]/(len(nodes_layer2)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    for j in range(len(nodes_layer3)):
        if nodes_layer3[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 500
        y = j*bitmap_dimension[1]/(len(nodes_layer3))+0.5*(bitmap_dimension[1]/(len(nodes_layer3)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    for node_layer2 in nodes_layer2:
        for node_layer1 in nodes_layer1:
            if (node_layer1.get_node_num(),node_layer2.get_node_num()) in conns \
                and conns[(node_layer1.get_node_num(),node_layer2.get_node_num())].get_status()==1:
#                print 'connection exists'
                start_coord = node_coords[node_layer1.get_node_num()]
                end_coord = node_coords[node_layer2.get_node_num()]
                draw.line((start_coord,end_coord), fill='black', width=1)

    for node_layer3 in nodes_layer3:
        for node_layer2 in nodes_layer2:
            if (node_layer2.get_node_num(),node_layer3.get_node_num()) in conns \
            and conns[(node_layer2.get_node_num(),node_layer3.get_node_num())].get_status()==1:
#                print 'connection exists'
                start_coord = node_coords[node_layer2.get_node_num()]
                end_coord = node_coords[node_layer3.get_node_num()]
                draw.line((start_coord,end_coord), fill='black', width=1)

    for j in range(len(nodes_layer1)):
        if nodes_layer1[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 100
        y = j*bitmap_dimension[1]/(len(nodes_layer1))+0.5*(bitmap_dimension[1]/(len(nodes_layer1)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    for j in range(len(nodes_layer2)):
        if nodes_layer2[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 300
        y = j*bitmap_dimension[1]/(len(nodes_layer2))+0.5*(bitmap_dimension[1]/(len(nodes_layer2)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    for j in range(len(nodes_layer3)):
        if nodes_layer3[j].get_status() == 1:
            color = 'red'
        else:
            color = 'black'
        x = 500
        y = j*bitmap_dimension[1]/(len(nodes_layer3))+0.5*(bitmap_dimension[1]/(len(nodes_layer3)))
        draw.ellipse((x-circlesize,y-circlesize, x+circlesize, y+circlesize), color)
        node_coords.append((x,y))

    im.save('deepnet-random.bmp')
    #im.show() # - You can add this command if you want a separate window to pop up \
        # with the visualized net output

Now we have written the function to visualize the net, let's call it from our main code and see what happens. To do this, I've placed some code in the file neural_net_tutorial1.py after the function definitions that we introduced above, so that when the Python file is run, our functions will be called. The code to construct the net, randomize it, and display the output as a .bmp is as follows:

Code Snippet : Add net construction, visualization and randomization to main routine

nodes_layer1, nodes_layer2, nodes_layer3, number_nodes_layer_1, number_nodes_layer_2, \
number_nodes_layer_3, threshold_activation, dim_x_viz, dim_y_viz, dim_x_viz_big, \
dim_y_viz_big, nodes, conns, total_number_nodes, total_number_conns, \
layer1_index, layer2_index, layer3_index = load_net_parameters()

randomize_net(conns)

visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns)

The output from this code should be a bitmap image. You can uncomment the command im.show() after the im.save() (found in the in visualization function) if you would like the visualization output to pop up in a separate window (tmp file). Personally I prefer to save the file so I know where it is. If you save the file, and then preview the saved .bmp in something like Windows Explorer, it should update automatically when you re-run the code.

Figure 2. The neural net after running the randomize_net() function.

Now that we have constructed the net, tested its operation and written a routine to graphically monitor the behaviour, we will start making the net do something useful.

SECTION 2

2.1 - Activating layer 1

First let's see if we can get the neurons (nodes) in the layers to fire, just like in a biological brain. We'll send in some random inputs at the lowest layer and then watch them activate nodes in the higher layers. For this we'll need to get the activation function (as described in section 1) working. The activation function determines the dynamics of the network, and in this example will be a very simple one. A node in a higher layer is just set to respond (become active) if there is a threshold number of active nodes which are connected to it from the lower layers. For example, if the threshold is set to 3 and there are 3 nodes in layer 1 which all connect to a node in layer 2 then this node will become active. You can imagine much more complex activation functions but for now we're going to keep things simple.

Let's add the layer 1 activation function. We already have a function called randomize_net - which currently randomizes the connections, but sets all the input nodes to status [0] (inactive). Let's extend this function so that the input layer activation is randomized as well as the connections.

Code Snippet : Adding feature to randomize_net() function to allow for random input node activation

def randomize_net(nodes_layer1, conns): # TUTORIAL2 - Add nodes_layer1

    for i in range(len(nodes_layer1)): # TUTORIAL2 - Add loop to randomize inputs
        if random.choice([0,1]):
            nodes_layer1[i].set_status(1)
        else:
            nodes_layer1[i].set_status(0)

    randomized_connections = []
    for i, j in conns.iteritems():
        if random.choice([0,1]):
            j.set_status(1)
            randomized_connections.append(1)
        else:
            j.set_status(0) #Set to 0 for randomly connected net, 1 for a fully connected net
            randomized_connections.append(0)
    return randomized_connections


    

2.2 - Activating the higher layers

Now that we have a function written which activates the nodes in layer 1 randomly, we need to write some new functions to update the higher layers (layers 2 and 3), so that the incoming signal 'propagates' up the network from layer 1 -> layer 2 -> layer 3. In load_net_parameters() we already set a variable called threshold_activation, which we will now use. The first function, update status, is our activation function. This tells the network to turn on a node if the counter value passed to it is greater than threshold_activation. You can imagine other activation functions, such as one where a node becomes active only if there is an EXACT value of active input nodes feeding into it, i.e. input == threshold_activation.

Code Snippet : Updating the status of the higher net layers using activation function

def update_status(node, input, threshold_activation):
    if input > threshold_activation:
        node.set_status(1)
    else:
        node.set_status(0)

def update_layer2(nodes_layer1, nodes_layer2, conns, threshold_activation):
    for node_layer2 in nodes_layer2:
        counter = 0
        for node_layer1 in nodes_layer1:
            if (node_layer1.get_node_num(),node_layer2.get_node_num()) in conns \
            and conns[(node_layer1.get_node_num(),node_layer2.get_node_num())].get_status()==1:
                if node_layer1.get_status() == 1:
                    counter +=1
        update_status(node_layer2, counter, threshold_activation)

def update_layer3(nodes_layer2, nodes_layer3, conns, threshold_activation):
    for node_layer3 in nodes_layer3:
        counter = 0
        for node_layer2 in nodes_layer2:
            if (node_layer2.get_node_num(),node_layer3.get_node_num()) in conns \
            and conns[(node_layer2.get_node_num(),node_layer3.get_node_num())].get_status()==1:
                if node_layer2.get_status() == 1:
                    counter +=1
        update_status(node_layer3, counter, threshold_activation)

2.3 - Back to the main routine

Now if you place the new function calls into the main code (as shown below) you should find that nodes in the higher layers are also becoming active. You might have to re-run the code a couple of times and look at the output .bmp as the nodes are set at random. You can also change activation_threshold to see how it affects the overall activation of the network. Try setting it to 2, or 0.

Code Snippet : Adding function calls to the main routine to update the net with activation added

nodes_layer1, nodes_layer2, nodes_layer3, number_nodes_layer_1, number_nodes_layer_2, \
           number_nodes_layer_3, threshold_activation, nodes, conns, total_number_nodes, total_number_conns, \
           layer1_index, layer2_index, layer3_index = load_net_parameters()

randomize_net(nodes_layer1, conns)
update_layer2(nodes_layer1, nodes_layer2, conns, threshold_activation)
update_layer3(nodes_layer2, nodes_layer3, conns, threshold_activation)
visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns)

Figure 3. The neural net after running the randomize_net() and updating the status of the higher levels to carry the activation of the nodes up the network.

SECTION 3

3.1 - Input data

Now that the network is responding, we can start to send it real data instead of random inputs. We want our network to be able to recognize and classify images, so let's start by looking at some image data. Here are some examples of the raw images that we'd like the network to classify:

Figure 4. Examples of training images

These are examples of images with the following labels (from left to right): Geordie, MukMuk, Suz, Apple. The images are taken from a video stream and the imageset we will use in this tutorial is the same one that was used in the QUFL tutorial.

To begin this part of the tutorial we'll need to import a few extra things into our code, so our header at the top of the Python file should now look like the one shown in the code snippet below. We have added the import of dwave_sapi, as this will be needed for the image compression routines which we will be exploring in this section. There are a few additional parameters necessary for using the D-Wave developer pack, such as the solver type and connection handle, and blackbox_parameter_compress, which sets the speed/accuracy tradeoff of the BlackBox routine. It is currently set to 1, which is the lowest value (BlackBox should run as fast as possible).

Code Snippet : Adding extra imports to handle the image data and compression routines

from PIL import Image, ImageDraw
import random
import os
import cPickle
from compressing_an_image import compress_me
from dwave_sapi import local_connection

solver = local_connection.get_solver("c4-sw_sample")
blackbox_parameter_compress = 1 # For compressing data


    

3.2 - Feature extraction background

We would like to compress the images into a useful form before sending them into the neural network. More specifically, the incoming images need to be converted into some representation where they can be expressed as a pattern in the input layer of nodes. To do this the incoming image is first reconstructed as a set of basis features using a separate algorithm (dictionary learning and reconstruction algorithm), so that the input layer nodes represent the learned dictionary, and nodes corresponding to each element of the dictionary light up if a feature is included in the reconstruction. Here are the dictionary atoms (K=8) that we discovered in the QUFL tutorial:

Figure 5. Dictionary atoms discovered in the QUFL tutorial

You don't have to use a dictionary here - you could use things like threshold filters, or Gabor filters or something similar to provide the input at the lowest level of the net. [As an aside, if you like thinking about biological neural networks, the feature extraction step is the hardcoded part of the information processing, similar to that performed by the human visual system. Light coming into our eyes undergoes a data transformation / compression step before being sent to the brain.]

3.3 - Output encoding

We're going to feed training data into the network during whilst the system finds the best configuration of its connections. How can we assess how well the net is doing during this process? Well, we're going to use a supervised learning approach. In supervised learning, the system knows that it got the answer right or wrong because we give it the correct answer so that it can learn from its mistakes. In order to do this we need a 'right answer' to accompany each piece of training data. These answers are known as the labels, and in this example we are going to fix the labels to be as follows:

\[ GR \to [1,0,0,0], \hspace{5mm} MukMuk \to [0,1,0,0],\hspace{5mm} suz \to [0,0,1,0],\hspace{5mm} Apple \to [0,0,0,1] \]

We save these in a structure known as actual_labels_list:

actual_labels_list = [[1,0,0,0], [0,1,0,0], [0,0,1,0], [0,0,0,1]]

3.4 - Preparing parameters for training and test data

In order to use the code that we developed in the previous tutorial, we'll need to write quite a large batch of new python functions. Don't worry about running the code until we get to the end of section 3 when all these functions have been explained. For now just concentrate on understanding what the functions do, and adding them to your main Python code.

First let's set up some code that handles all the filesystem bits and pieces. We'll call this training_data_setup.py.

The size of the images shown in Figure 4 are 80x112 pixels, and the size of the tiny versions of them that we'll get the code to put on our deepnet.bmp will be 23x33, so those parameters go into the code:

Code Snippet : Writing the training_data_setup() function

def training_data_setup():

    dim_x_viz = 23 #For the visualization, sets the size of the little images
    dim_y_viz = 33
    dim_x_viz_big = 80 # The larger versions of the same dictionary atoms
    dim_y_viz_big = 112

    # Load the dictionary ------------------------------------------------------------------------
    cwd = os.getcwd()
    filepath = "../images-for-QUFL"

    filename_to_read_final_D_values = cwd + "\DX.txt"
    mypicklefile = open(filename_to_read_final_D_values, 'r')
    D = cPickle.load(mypicklefile)
    mypicklefile.close()

    filename_to_read_mean_pixel_values = cwd + "\mean_pixels.txt"
    mypicklefile = open(filename_to_read_mean_pixel_values, 'r')
    mean_value_of_raw_pixels = cPickle.load(mypicklefile)
    mypicklefile.close()

    #Load the training and test data from the txt files ------------------------------------------
    numbers_train = ['000', '010', '020', '030', '040', '050', '060', '070', '080', '090', '100', '110', \
                     '120', '130', '140', '150', '160', '170', '180', '190', '200']
    numbers_test = ['005', '015', '025', '035', '045', '055', '065', '075', '085', '095', '105', '115', \
                    '125', '135', '145', '155', '165', '175', '185', '195', '205']

    folders = ['GR', 'MukMuk', 'suz', 'Apple']
    actual_labels_list = [[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]
    vector_of_twos = [2,2,2,2]
    numpixels = dim_x_viz_big*dim_y_viz_big; N_raw = numpixels * 3
    K = 8

    return dim_x_viz, dim_y_viz, dim_x_viz_big, dim_y_viz_big, filepath, cwd, D, folders, \
           numbers_train, numbers_test, actual_labels_list, vector_of_twos, numpixels, \
           mean_value_of_raw_pixels, N_raw, K


    

3.5 - Adding training data to the visualization

Next we will write a function that turns the dictionary (currently stored as a text file called DX.txt) into a picture so we can look at the features:

Code Snippet : Turning the text file DX.txt into an image, K8_dict.bmp

def turn_D_into_a_picture(D, K, number_of_pixels_in_input_images_x, number_of_pixels_in_input_images_y):

    mypicklefile = open('mean_pixels.txt', 'r')
    mean_value_of_raw_pixels = cPickle.load(mypicklefile)
    mypicklefile.close()
    dictionary_atoms=Image.new('RGB',(number_of_pixels_in_input_images_x*K,number_of_pixels_in_input_images_y))

    for y_pixels in range(number_of_pixels_in_input_images_y):
        for x_pixels in range(number_of_pixels_in_input_images_x):
            for image_number in range(K):
                pixelnumber=x_pixels+y_pixels*number_of_pixels_in_input_images_x
                pixel_value=(D[pixelnumber*3,image_number]+mean_value_of_raw_pixels[pixelnumber*3,0], \
                             D[pixelnumber*3+1,image_number]+mean_value_of_raw_pixels[pixelnumber*3+1,0], \
                             D[pixelnumber*3+2,image_number]+mean_value_of_raw_pixels[pixelnumber*3+2,0])
                dictionary_atoms.putpixel((x_pixels+number_of_pixels_in_input_images_x*\
                                                    image_number,y_pixels), pixel_value)
    dictionary_atoms.save('K8_dict.bmp')

The image K8_dict.bmp should look identical to figure 5. These images are the same size as the training examples that are sent into the network. However, one thing we want to be able to do is put these features onto our deepnet.bmp, and at the moment they are much too big, so we will resize them. here is a code snippet which resizes the dictionary assuming you have already generated K8_dict.bmp:

Code Snippet : Resizing the large dictionary so that we have small icons available to put on deepnet.bmp

def resize_the_K8_dict(dim_x_viz, dim_y_viz):
    im = Image.open('K8_dict.bmp')
    im2 = Image.new('RGB', (dim_x_viz*8, dim_y_viz))
    im2.save('resized_K8_dict.bmp')
    imResize = im.resize((dim_x_viz*8,dim_y_viz))
    imResize.save('resized_K8_dict.bmp')

The result from running this code should be a smaller version of K8_dict, which we call K8_resized.bmp. Don't worry about running these function yet, as we will run them all later on and look at the outputs.

Next we write a function that allows us to put those little resize icons onto the neural net image:

Code Snippet : Putting the newly resized dictionary atoms on the deepnet.bmp image:

def put_atoms_from_K8_dict_on_net(w_vector, dim_x_viz, dim_y_viz):
    im_dict = Image.open('resized_K8_dict.bmp')
    y_offset = 17
    raw_data_dict = list(im_dict.getdata())
    im = Image.open('deepnet.bmp')
    im_num = len(w_vector)
    for i in range(im_num):
        if w_vector[i]:
            for x in range(0,dim_x_viz):
                for y in range(0,dim_y_viz):
                    im.putpixel((x,((y_offset*i)+y+(i*dim_y_viz))), \
                                raw_data_dict[(i*dim_x_viz)+x+y*(im_num*dim_x_viz)])
    im.save('deepnet.bmp')

We also want a visualization routine to know what the correct label SHOULD be, so we'll write that now:

Code Snippet : Adding a green dot to show what the ACTUAL label associated with each datapoint is:

def put_correct_label_on_net(folder):
    im = Image.open('deepnet.bmp')
    draw = ImageDraw.Draw(im)
    circlesize = 6
    y_offsets = [50,150,250,350]
    draw.ellipse((550-circlesize,y_offsets[folder]-circlesize, 550+circlesize, \
                  y_offsets[folder]+circlesize), 'green')
    im.save('deepnet.bmp')

3.6 - Loading training and test data into the main code

Now we get onto the main part of this section. We now need to write code takes all the input image from the imageset and converts them it into a series of column w_vectors of size K, for example:

w_i =
[[0],
[1],
[1],
[0],
[0],
[0],
[0],
[0]]

Which corresponds to the activation of the nodes at the input layer of the neural net. We want to do this in advance and save them to a file, because generating a w_i for each image is in itself an optimization procedure which uses BlackBox, (as we discovered in the QUFL tutorial). So we want to compress the data only once to save time.

Code Snippet : Compressing the training and test data

def compress_training_and_test_data(filepath, folders, numbers_train, numbers_test, D, \
                                    mean_value_of_raw_pixels, numpixels, N_raw, \
                                    K, blackbox_parameter_compress):
    training_data_outer = []
    test_data_outer = []
    use_blackbox = 0
    solver_flag = 0

    for j in range(len(folders)):
        training_data_inner = []
        for i in range(len(numbers_train)):
            base_file_path = filepath+'/'+folders[j]+'/image'+numbers_train[i]+'.bmp'
            print 'compressing training datapoint', base_file_path
            answer = compress_me(base_file_path, D, mean_value_of_raw_pixels, numpixels, N_raw, \
                                 K, use_blackbox, solver_flag, blackbox_parameter_compress)
            training_data_inner.append(answer)
        training_data_outer.append(training_data_inner)

    for j in range(len(folders)):
        test_data_inner = []
        for i in range(len(numbers_test)):
            base_file_path = filepath+'/'+folders[j]+'/image'+numbers_test[i]+'.bmp'
            print 'compressing test datapoint', base_file_path
            answer = compress_me(base_file_path, D, mean_value_of_raw_pixels, numpixels, N_raw, \
                                 K, use_blackbox, solver_flag, blackbox_parameter_compress)
            test_data_inner.append(answer)
        test_data_outer.append(test_data_inner)

    mypicklefile = open('training_data_w_vectors.txt', 'w')
    cPickle.dump(training_data_outer, mypicklefile)
    mypicklefile.close()
    mypicklefile = open('test_data_w_vectors.txt', 'w')
    cPickle.dump(test_data_outer, mypicklefile)
    mypicklefile.close()

    raw_input('Training and test data compressed. Continue?')

Let's now load the compressed data that we wrote to 2 pickle files into our main code body. We'll write another function to do this:

Code Snippet : Loading the compressed data (w_vectors) into our main code.

def load_training_and_test_data():

    mypicklefile = open('training_data_w_vectors.txt', 'r')
    training_data_outer = cPickle.load(mypicklefile)
    mypicklefile.close()

    mypicklefile = open('test_data_w_vectors.txt', 'r')
    test_data_outer = cPickle.load(mypicklefile)
    mypicklefile.close()

    return training_data_outer, test_data_outer

We also need a new function that takes in a w_vector from our loaded training data and uses it to initialize the net. So let's write yet another function to do that. You can think of this function as being very similar to the part of randomize_net() that set the input nodes, but now instead of setting them randomly we will set them according to an input image's w_vector.

Code Snippet : Initializing the net by giving it a w_vector as input.

def initialize_net(nodes_layer1, conns, w_vector, connections_list):
    for i in range(len(nodes_layer1)):
        if i < len(w_vector):
            nodes_layer1[i].set_status(w_vector[i])
        else:
            nodes_layer1[i].set_status(0)
    counter = 0
    for i, j in conns.iteritems():
        j.set_status(connections_list[counter])
        counter +=1

3.7 - Sending an image to the network

The final function that we need to write in this section is something to pass an image of our choice to the network. Here is the function. It asks the user which image they would like to use by first asking which folder they wish the image to come from, and then asks for the index of the image. Note that because we only chose select images for our testing and training set, the index of the image specified must be below len(test_data_outer)

Code Snippet : Allowing a user to send an image through the net and generate a prediction at the output layer

def send_an_image_through_net(nodes_layer1, nodes_layer2, nodes_layer3, \
                              conns, threshold_activation, dim_x_viz, dim_y_viz):

    input = raw_input('Assign labels to images? Hit enter')
    while input == '':
        mypicklefile = open('test_data_w_vectors.txt', 'r')
        test_data_outer = cPickle.load(mypicklefile)
        mypicklefile.close()

        mypicklefile = open('neural-net-conns-random.txt', 'r')
        best_net = cPickle.load(mypicklefile)
        mypicklefile.close()

        print 'Enter folder of the image to process, in range 0 to', len(folders)-1, ':'
        folder = int(raw_input('folder?'))
        print 'Enter number of the image to process, in range 0 to', len(test_data_outer[0])-1, ':'
        number = int(raw_input('number?'))
        test_data_w = test_data_outer[int(folder)][int(number)]

        bit_string_column_vector = test_data_w
        w_vector = []
        for datum in bit_string_column_vector:
        # Converts from a column vector to a row vector
                raw_value = datum[0]
                w_vector.append(raw_value)

        initialize_net(nodes_layer1, conns, w_vector, best_net)
        update_layer2(nodes_layer1, nodes_layer2, conns, threshold_activation)
        update_layer3(nodes_layer2, nodes_layer3, conns, threshold_activation)
        visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns)
        put_atoms_from_K8_dict_on_net(w_vector, dim_x_viz, dim_y_viz)
        put_correct_label_on_net(folder)
        input = raw_input('Again? Press enter. Type "q" and enter to quit')


        

3.8 - Back to the main routine

Now that we have written all the functions for Section 3, we can get back to calling them from our main routine, as shown in the code snippet below. First we import all the parameters that we defined earlier, plus some new ones - training_data_setup.py and load_training_and_test_data(). Note that we have to run the compress_training_and_tset_data() before we can load_training_and_test_data(). Once we have compressed the data and loaded it into the program, we can then create the image files that will be used to put small versions of the features on the deepnet.bmp using the function turn_D_into_a_picture() and resize_the_K8_dict().

The outputs from running the resize_the_K8_dict() code is shown in the figure below. This won't actually pop up, but it will save an image K8_dict_resized.bmp to your harddrive in the same folder as the code, which looks something like this:

Figure 6. The output from resize_the_K8_dict()

After compressing data and generating image icons, all that remains is to randomize the net as we did earlier, and try sending an image through the net!

Code Snippet : Adding several calls to the new functions to the main routine.

nodes_layer1, nodes_layer2, nodes_layer3, number_nodes_layer_1, number_nodes_layer_2, \
        number_nodes_layer_3, threshold_activation, nodes, conns, total_number_nodes, \
        total_number_conns, layer1_index, layer2_index, layer3_index = load_net_parameters()

dim_x_viz, dim_y_viz, dim_x_viz_big, dim_y_viz_big, filepath, cwd, D, folders, \
           numbers_train, numbers_test, actual_labels_list, vector_of_twos, numpixels, \
           mean_value_of_raw_pixels, N_raw, K = training_data_setup()

#Next line only needs to run once:
compress_training_and_test_data(filepath, folders, numbers_train, numbers_test, D, \
                                mean_value_of_raw_pixels, numpixels, N_raw, K, blackbox_parameter_compress)

training_data_outer, test_data_outer = load_training_and_test_data()

#Next line only needs to run once:
turn_D_into_a_picture(D, K, dim_x_viz_big, dim_y_viz_big)

#Next line only needs to run once:
resize_the_K8_dict(dim_x_viz,dim_y_viz)

randomized_connections = randomize_net3(nodes_layer1, conns)

mypicklefile = open('neural-net-conns-random.txt', 'w')
cPickle.dump(randomized_connections, mypicklefile)
mypicklefile.close()

send_an_image_through_net(nodes_layer1, nodes_layer2, nodes_layer3, conns, \
                          threshold_activation, dim_x_viz, dim_y_viz)

Note that we save the randomized connections to a file. This is because we do not want to re-randomize the net everytime we send an image through it, or it would be difficult to tell if it is working properly. If you want to re-randomize the net you can always quit the send_an_image_through_net() subroutine and run the code neural-net-tutorial3.py from the beginning again. Sometimes the net may randomize in a way that means higher level nodes do not activate, so you may want to try re-running the code a few times to see how differently randomized nets affect the output.

Aldo note that you can comment out the compress_training_and_test_data(), turn_D_into_a_picture(), resize_the_K8_dict functions after you have run them once, to save time. You don't need to recreate the compressed representations or the image icons every time you run the program. This feature is highlighted with some comments in the code snippet above.

Observe how many times the net correctly predicts the output (red dot matches the green dot) when the connections are randomized like this... it isn't very often! We are going to fix that in the next step as we are now going to start training the net using the quantum solvers.

Upon running the main routine and following the user prompts to enter folder number and image number, you should see something like this:

Figure 7. Screengrab for sending an image of user's choice through the neural net

SECTION 4

4.1 - Crafting an objective function

Now what we are trying to do is optimize the connections between the nodes in the network such that the activation at the top layer matches labels possessed by the input images. The network will learn this mapping by minimizing an objective function that captures the error between predicted labels (at the output layer) and actual labels (supplied with the training data). So we want the network to calculate the errors that it makes when it predicts what an incoming image is. Here are examples of the network making incorrect predictions and the associated 'errors' in each case:

Figure 8a. Incorrect net prediction: the network makes 2 errors

Figure 8b. Incorrect net prediction: the network makes 3 errors

Figure 8c. Incorrect net prediction: the network makes 1 error

The objective function for this application is very simple. We will just try to minimize the overall error that the network makes. You can think of some more complicated objective functions, for example you could also add another term which penalizes the network for turning on the wrong number of labels which is separate from the squared error loss.

The squared error for an individual training example is given by:

\[ L(x, C) = [f_{net}(\bar{C}_{ij},\bar{w}_x,act)-\bar{y}_d]^2 \]

So our loss function is just the sum of all such errors for every example, \( d \), in the training set:

\[ L(C) = \sum_{d = 1}^D [f_{net}(\bar{C}_{ij},\bar{w}_d,act)-\bar{y}_d]^2 \]

We want to minimize the loss over all training examples by adjusting the connections, C. The minimization of L over the set of connections, C, is the optimization problem we are trying to solve:

\[ argmin(C) \sum_{d = 1}^D [f_{net}(\bar{C}_{ij},\bar{w}_d,act)-\bar{y}_d]^2 \]

What this basically means is that for each training data example, we take the difference between the predicted 4 labels and the actual 4 labels and square the result. In the three examples above, \(L(x, C) \) would be equal to 4, 9 and 1 respectively. We then add up all the square errors to contribute to the final error that the network makes. So now that we have our objective function we can begin to write the routine that trains the network.

4.2 - Training the neural net

First let's make sure we have a parameter for BlackBox specified. I've explicitly made it different from the parameter that is used by BlackBox to do the image reconstruction part, so we can change these separately. This should go in the top of the file with the other import statements, as shown in the code snippet below. A couple other imports have been added, to handle the calculation of the objective function

Code Snippet : Adding the BlackBox parameter for net training to the top of the file

from PIL import Image, ImageDraw
import random
import os, operator # Add operator for tutorial 4
import cPickle
from compressing_an_image import compress_me
from dwave_sapi import local_connection, BlackBoxSolver # Add BlackBox solver for tutorial 4
from numpy import array # Add array for tutorial 4

solver = local_connection.get_solver("c4-sw_sample")
blackbox_parameter_compress = 1 # For compressing data
blackbox_parameter = 1 # Add for tutorial 4 - this parameter sets BlackBox for training the net.

Readers familiar with the BlackBox tutorial will recall that the function that you want to minimize is now sent into BlackBox in a standard way by creating a class object to hold the iteratively called function. In practice this means that you will need to calculate the objective function - i.e. the difference between the predicted labels and the actual labels for all training points - inside the class definition. The next code snippet shows how to do this.

Code Snippet : Class for holding the BlackBox function

class ObjClass(object):

    def __call__(self, states, numStates):

        states_bin  = [(item+1)/2 for item in states]
        # converting to 0/1 from -1/+1
        stateLen = len(states)/numStates
        # this is the length of each individual state; this should be equal to K

        ret = []
        for state_number in range(numStates):
            optimized_connections_list = array(states_bin[state_number*stateLen:(state_number+1)*stateLen])
            running_counter = 0
            for j in range(len(training_data_outer)):
                for dataset in training_data_outer[j]:
                    bit_string_column_vector = dataset
                    w_vector = []

                    for datum in bit_string_column_vector:
                    #Converts from a column vector to a row vector
                        raw_value = datum[0]
                        w_vector.append(raw_value)

                    initialize_net(nodes_layer1, conns, w_vector, optimized_connections_list)
                    update_layer2(nodes_layer1, nodes_layer2, conns, threshold_activation)
                    update_layer3(nodes_layer2, nodes_layer3, conns, threshold_activation)
                    actual_labels = actual_labels_list[j]
                    predicted_labels = []

                    predicted_labels.append(nodes_layer3[0].get_status())
                    predicted_labels.append(nodes_layer3[1].get_status())
                    predicted_labels.append(nodes_layer3[2].get_status())
                    predicted_labels.append(nodes_layer3[3].get_status())

                    label_errors = map(operator.sub, actual_labels, predicted_labels)
                    label_errors_square = map(operator.pow, label_errors, vector_of_twos)
                    label_errors_square_sum = sum(label_errors_square)
                    running_counter +=label_errors_square_sum

            result = running_counter
            ret.append(result)
        return tuple(ret)

num_vars = len(conns)
obj = ObjClass()

4.3 - Back to the main routine

Once we have written this class, we can now call it by incorporating some code into our main routine as follows. I've included some printout so we know what is going on. Note that we have also now removed the functions compress_training_and_test_data(), turn_D_into_a_picture(), and resize_the_K8_dict() from the main routine, as we should already have the output from these functions saved to disk.

Code Snippet : Calling BlackBox and saving the answer it finds to a file

nodes_layer1, nodes_layer2, nodes_layer3, number_nodes_layer_1, number_nodes_layer_2, \
        number_nodes_layer_3, threshold_activation, nodes, conns, total_number_nodes, \
        total_number_conns, layer1_index, layer2_index, layer3_index = load_net_parameters()

dim_x_viz, dim_y_viz, dim_x_viz_big, dim_y_viz_big, filepath, cwd, D, folders, \
           numbers_train, numbers_test, actual_labels_list, vector_of_twos, numpixels, \
           mean_value_of_raw_pixels, N_raw, K = training_data_setup()

training_data_outer, test_data_outer = load_training_and_test_data()

num_vars = len(conns)
obj = ObjClass()

#Some printout so we know what the network is using to train on
print 'number of training points per folder = ', len(training_data_outer[0])
print 'number of test points per folder = ', len(test_data_outer[0])
print 'total number of training and test points =', \
    (len(training_data_outer[0])+len(test_data_outer[0]))*len(folders)
print 'training network... please wait...'

#Here is the main BlackBox routine
blackbox_solver = BlackBoxSolver(solver)
blackbox_answer = blackbox_solver.solve(obj, num_vars, cluster_num = 2, min_iter_inner = \
    blackbox_parameter, max_iter_outer= blackbox_parameter, unchanged_threshold=blackbox_parameter, \
    max_unchanged_objective_outer=blackbox_parameter, \
    max_unchanged_objective_inner = blackbox_parameter, \
    unchanged_best_threshold = blackbox_parameter, verbose=1)

print blackbox_answer
blackbox_answer_bin = [(item+1)/2 for item in blackbox_answer] # converting to 0/1 from -1/+1

print "The best bit string we found was:",blackbox_answer_bin

# Save the best network that BlackBox found to a file so you don't have to rerun the optimization
# each time to get at the answer.
print 'saving net to file...'
mypicklefile = open('trained-neural-net-conns.txt', 'w')
cPickle.dump(blackbox_answer_bin, mypicklefile)
mypicklefile.close()

visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns)
send_an_image_through_net(nodes_layer1, nodes_layer2, nodes_layer3, conns, \
                          threshold_activation, dim_x_viz, dim_y_viz)

As the BlackBox code runs through, it will output several iterations as it searches the space to find the best configuration of synapses. This can anything from a few seconds to several minutes depending upon the spec of your computer, so be patient during this step. We included a handy line with some printout that informs the user that BlackBox is thinking...

The code should eventually return a bitstring found by BlackBox, which corresponds to the best subset of connections discovered. We save this bitstring to a file called 'trained-neural-net-conns.txt' as the training routine can sometimes take a long time, and we want to keep any good answer that BlackBox found after the program terminates.

You should see something similar to the following image. Again I have shown the deepnet.bmp on top of the code printout to show how the send_an_image_through_net() is now working with the newly discovered connections. For this run I had threshold-activation = 0.

Figure 8. Running the BlackBox code to discover an optimal connection structure

SECTION 5

5.1 - Testing the net performance using test data

The last thing we need to do is to test how well our trained network is performing. To do this we need some test data. Luckily our compress_training_and_test_data has already provided us with the variable test_data_outer, so all we need to do is compute the error between the actual labels (again assigned by what 'folder' they are in) and the predicted labels coming from the network now that we have trained it.

Also added to this routine is a few lines of code to calculate the error of the neural net when considered as an image classifier and the sparsity of the network (number of connections formed as a percentage of all possible synaptic connections).

Code Snippet : Running the test data to investigate the suitability of the neural net as a classifier

def test_network():
    running_counter = 0
    for j in range(len(test_data_outer)):
        for dataset in test_data_outer[j]:
            bit_string_column_vector_test = dataset
            w_vector_test = []

            for datum in bit_string_column_vector_test:
            #Converts from a column vector to a row vector
                raw_value = datum[0]
                w_vector_test.append(raw_value)

            initialize_net(nodes_layer1, conns, w_vector_test, blackbox_answer_bin)
            update_layer2(nodes_layer1, nodes_layer2, conns, threshold_activation)
            update_layer3(nodes_layer2, nodes_layer3, conns, threshold_activation)

            actual_labels_test = actual_labels_list[j]
            predicted_labels_test = []
            predicted_labels_test.append(nodes_layer3[0].get_status())
            predicted_labels_test.append(nodes_layer3[1].get_status())
            predicted_labels_test.append(nodes_layer3[2].get_status())
            predicted_labels_test.append(nodes_layer3[3].get_status())

            label_errors_test = map(operator.sub, actual_labels_test, predicted_labels_test)
            label_errors_test_square = map(operator.pow, label_errors_test, vector_of_twos)
            label_errors_square_sum = sum(label_errors_test_square)
            running_counter +=label_errors_square_sum

    print 'total number of errors made on test set = ', running_counter
    print 'total number of labels assigned on test set = ', len(folders)*len(numbers_test)*4

    #Count up the connections
    counter = 0
    for node_layer2 in nodes_layer2:
        for node_layer1 in nodes_layer1:
            if (node_layer1.get_node_num(),node_layer2.get_node_num()) in conns \
                and conns[(node_layer1.get_node_num(),node_layer2.get_node_num())].get_status()==1:
                counter +=1
    for node_layer3 in nodes_layer3:
        for node_layer2 in nodes_layer2:
            if (node_layer2.get_node_num(),node_layer3.get_node_num()) in conns \
            and conns[(node_layer2.get_node_num(),node_layer3.get_node_num())].get_status()==1:
                counter +=1

    print ''
    print 'number of connections in final net', counter
    print ''
    print 'Sparsity of learned network =', \
    counter*100/(number_nodes_layer_1*number_nodes_layer_2+\
                 number_nodes_layer_2*number_nodes_layer_3), '%'
    print ''
    print '*******************************'
    print 'classifier error = ', running_counter*100/(len(folders)*len(numbers_test)*4), '%'
    # 4 labels per test image - it has to get all 4 labels right
    print '*******************************'
    print ''

Given that the trained network connections list is also saved to a file as explained earlier, this means that you can comment out the training part of the net, i.e. the lines referencing the BlackBox solver in the main routine whilst we perform the testing step if you so wish, and instead load in the trained net from the file, as shown below.

Insert the next code snippet into your main routine after the BlackBox section (at the end) and run the code.

We've also re-added the send_an_image_through_network() at the end of this code, so that you can see how well the network is performing on individual examples that you send through now that it has been trained. Compare this with the results from randomly setting the connections... there is quite a difference!

Code Snippet : Running the test data through the network to see how well it performs

mypicklefile = open('trained-neural-net-conns.txt', 'r')
blackbox_answer_bin = cPickle.load(mypicklefile); mypicklefile.close()

test_network()

visualizeNet(nodes_layer1, nodes_layer2, nodes_layer3, conns)
send_an_image_through_net(nodes_layer1, nodes_layer2, nodes_layer3, conns, threshold_activation \
    , dim_x_viz, dim_y_viz)

Here is what the output should look like:

Figure 9. Running the test data to investigate the accuracy of the neural net as a classifier

That's it! Hope you enjoyed building a quantum brain. Believe it or not, although the code seems involved, this is a very simple neural net, and there are many extensions that could be added to make it more interesting and useful for classifying real data. Here are some ideas for extending the code:

A better objective function

The square loss is probably not the best objective function that we could use. Feel free to implement other objective functions that might capture a better way to learn from the errors made by the net.

Add regularization

The objective function doesn't try to find a sparse net, which can cause overfitting to the training data. Try adding a regularization term to the objective function which favours nets that are more sparsely connected .

Add more layers

Of course deep networks can be more than 3 layers! It would be interesting to investigate if the representational power (accuracy) of a network can be increased by adding more hidden layers.