Unsupervised Learning with Neural Networks

Manifold Learning

Create a synthetic two-dimensional manifold:
In[1]
Click for copyable input
Out[1]
Create a net with a "bottleneck" layer in order to learn a parameterization of the manifold:
In[2]
Click for copyable input
Out[2]
Create a loss network that computes a loss based on the "reconstruction error" on the input manifold:
In[3]
Click for copyable input
Out[3]
Train the loss network on the manifold and extract the original net from the loss network:
In[4]
Click for copyable input
Out[4]
Visualize how the net projects arbitrary points onto the manifold:
In[5]
Click for copyable input
Out[5]
Split the net into "encoder" and "decoder" networks (the encoder parameterizes points using a single scalar value, whereas the decoder reconstructs the point from this parameterization):
In[6]
Click for copyable input
Out[6]
Out[6]
Color each point in the original manifold by its parameterization under the encoder:
In[7]
Click for copyable input
Out[7]
Obtain the range of the parameterization by applying the encoder to the manifold:
In[8]
Click for copyable input
Out[8]
Show the reconstruction over this range along with the original manifold:
In[9]
Click for copyable input
Out[9]

MNIST Digit Autoencoder

First, obtain the training data and select images corresponding to digits 0 through 4:
In[1]
Click for copyable input
Out[1]

An autoencoder has the same shape for its input and output, and has a bottleneck in the middle of the net to prevent the net simply memorizing the inputs.

Create a net that takes an input with dimensions {1,28,28} and returns an output with the same dimensions {1,28,28}:
In[2]
Click for copyable input
Out[2]

The preceding net takes as input 784 real numbers and compresses this to a vector of 40 real numbers (the bottleneck). The net then needs to reconstruct the original image from these 40 real numbers.

Create an encoder that converts images to tensors of dimension {1,28,28}:
In[3]
Click for copyable input
Out[3]
Attach a loss to the autoencoder net that penalizes the input being different from the output:
In[4]
Click for copyable input
Out[4]
Train the network on the reconstruction error:
In[5]
Click for copyable input
Out[5]
Obtain a subnetwork that performs only reconstruction:
In[6]
Click for copyable input
Out[6]
Reconstruct some sample images:
In[7]
Click for copyable input
Out[7]
Obtain a subnetwork that produces the code vector:
In[8]
Click for copyable input
Out[8]
Compute codes for all of the test images:
In[9]
Click for copyable input
Project the code vectors to three dimensions and visualize them along with the original labels (not seen by the network). The digit classes tend to cluster together:
In[10]
Click for copyable input
Out[10]
Perform automatic clustering directly on the code vectors and show a sample taken from each cluster:
In[11]
Click for copyable input
Out[11]
Visualize a hierarchical clustering of random representatives from each class:
In[12]
Click for copyable input
Out[12]
Click for copyable input

Related Tutorials