Fixing the Magical Hat

• Hit go without making any changes.
• The background color slowly changes between blue and orange.
• Eventually, the blue points get a blue background, and the orange points get an orange background.

• The learning rate helps determine how quickly the network learns.
• A small learning rate results in careful, steady progress.
• But here the learning rate is much too slow.
  1. Hit the reset (circle) button.
  2. Find the learning rate drop down at the top of the screen, and select a slightly larger number.
  3. Press start (play).
• Try resetting and running the simulation with a range of different learning rates.

But be careful. Setting the learning rate too high in some of the later activities will cause the magical hat's training to jump around too much.

Next to the "play" button is a circle.

• Click this button to reset the training run.
• Try repeatedly pressing reset then play. Notice how the different training runs behave differently.
• Training a magical hat is like playing the lottery - some training runs work much better than others, even with the same settings!

Below the "Data" label is a button titled Regenerate.

• This moves around the blue and orange circles.
• If you press this button while training, the hat will work to adjust its predictions to match the new data.

At the far left of the screen, there is a button called "Regenerate".

• Click that button and see your dataset update.
• This shows the magical hat how it should assign a different group of students to their houses.

Let's try to see one of the problems that a big learning rate can cause.

• Set the Learning Rate to 10
• Press Go
• Press Regenerate a few times, waiting a little bit between each click

If you're lucky, you will start to see the line jumping around wildly.
In more complex datasets, a high learning rate will cause the magical hat to repeatedly over-correct its estimates. But don't worry if you don't see this behavior here - the training is random.

In the remainder of the exercises, stick with a learning rate of somewhere between 0.001 and 0.03, unless you want to see the training become unstable.

• At first, the hat does a very bad job of sorting.
• To improve, it must measure the error of its current prediction about each student's house.
• If you add up these errors, you get the current loss.
• The graph in the top right of the interative shows a picture of how the loss has changed during training.
  • You want the error (loss) to steadily go down.
  • A flat loss means that the model isn't learning anything,
  • An increasing loss means that the model is doing worse and worse!
  • When loss becomes noisy, the training is unstable, and the colors jump around.

• First, hit "Play". What kind of pattern does the hat learn?
• Is this pattern enough to separate the people inside the circle from the ones outside the circle?

• Look for 'Hidden Layer' in the middle of the screen.
• Click +, and observe as two new 'nodes' show up in the middle of the screen.
• Hit go, and see what happens.
• Try adding a second hidden layer, and hit play. What happens? What does a third, fourth, and fifth layer do?

• Go back to a single hidden layer (or refresh your page and add a single new layer).
• Above each layer, it says the number of neurons (nodes) in the layer, and has a +/- sign to increase and decrease the number of those nodes.
• Try adding a neuron to your layer, and hit play. What happens?
• Add additional layers and neurons. What are the impact of these changes?

The smallest network that can approximate this data set has a single hidden layer of three neurons.

• Try adding a second hidden layer, and hit play. What happens?
• What does a third, fourth, and fifth layer do?

• Go back to a single hidden layer of two neurons (or refresh your page and add a single new layer).
• Try adding a third neuron to your layer, and hit play. What happens?
• Now add a fourth neuron. What happens?
• Experiment with different numbers of layers and neurons in each layer.

The smallest network that can approximate this data set has a single hidden layer of four neurons.

With a few additional layers of four or more neurons, the network converges faster.

But with two many layers, the training does not converge at all.

• Try increasing the number of neurons in each layer, and try increasing the number of layers. Can you help the hat learn to separate the spirals?
• What if you increase the layers to 6 and neurons to 8/layer (the largest allowed)? We clearly need some more advanced magical tricks!

• Put the mouse over different edges. You will see a number pop up. This is called the weight of the edge.
• Blue means positive and orange means negative, with darker colors representing larger numbers.

• Most of what the magic hat does is multiply and add numbers.

• Each neuron also includes a more complicated function, called activation function.

• You can think of these as the magic behind the neuron.
• To train more complicated networks, it can be helpful to try chainging these magical brains.

• Try setting the number of neurons and layers to be very large, then press "go".

• It seems to work, but it jumps around a lot.

• Now look for a drop down list next to the word activation, and click on the word ReLU.

• Press "Go", and watch the network train.
• Often (but not always), the network will train faster and more smoothly.

It turns out that training the hat sometimes breaks when one neuron does an unfair amount of work. We need to divide up the magic more carefully using regularization. But different versions of regularization work for different problems. Think about fairness of division of work among group projects.

• Its not fair to make one person do all the work if everyone is equally good at everything. L2 regularization tries to make sure that one person isn’t being made to do all the work.
• If everyone in the group is good at something different, it will take less work if you divide up tasks, than if one person tries to do everything. L1 regularization tries to make sure that everyone is doing the task that they excel at.

• Find the drop down under the word Regularization and select L2.

• Depending on the number of layers and neurons you are using, you may need to wait some time for the training to converge.

• Keep an eye on the graph on the top right - the line tells you a rough measurement of the error that your network is making. A downward line tells you that the error is getting smaller, and the network is still learning, even if the network doesn't match the data set yet.

• Once you can train the hat to match the spiral, write down your settings.
  Then try reducing the number of layers and/or neurons to find the smallest network that will match the spiral.

You can train the hat to match the spiral fairly quickly with the following settings: - 5 hidden layers, with 6 neurons per layer (30 total neurons) - Learning rate: 0.03 - Activation: ReLU - Regularization: L2 - Regularization Rate: 0.001

But this is not the best solution. How small of a network can you train to match the spiral quickly?

Can you train make a smaller network if you are willing to wait longer? How much longer will you need to wait?