Neural Networks Notes

Lectures

1A

Machine learning is good for complicated processes combining weak rules to hit a moving target. The machine learning approach is to mine data for rules and it should work for new cases, not just the cases it was trained on.

Best for learning algorithms:

• Recognizing patterns
  • objects in real scenes
  • facial identities or expressions
  • spoken words
• Anomalies
  • unusual sequences of credit card transactions
  • unusual patterns of sensor readings in nuclear power plants
• Prediction
  • future stock prices
  • which movies a person will like

The MNIST database of hand-written digits is the machine learning equivalent of fruit flies to genetics. TIMIT is the same benchmark for speech recognition.

1B : Real Neurons

A cortical neuron has one axon that branches and dendritic tree that collects input form other neurons. A spike of activity in the axon causes charge to be injected into the post-synaptic neuron. Shen a spike of activity travels along an axon and arrives at a synapse it causes vesicles of transmitter chemicals to be released. There are several kinds of transmitter.

• Each neuron receives input from other neurons (a few neurons are also connect to receptors)
• The effect of each input line on the neuron is controlled by a synaptic weight.
• The synaptic weight adapts so that the whole network learns to perform useful computations.
• You have 10^11 neurons each with about 10^4 weights.

The cortex is modular. Different bits of the cortex do different things. Local damage to the brain has specific effects. But the cortex looks pretty much the same all over. This suggests that there is a uniform learning pattern at work.

1C : Simple models of neurons

To model things we have to idealize them.

Linear Neuron

Simple but misleading.

y = b + Σ x_iw_i

• b: bias
• w: weight
• x: input

The graph is a straight line

Binary Threshold Neurons

First compute a weighted sum of the inputs. If the weighted sum exceeds a threshold, send out a spike.

Graph is a cliff.

Rectified Linear Neurons

Sometimes called linear threshold neurons. Compute a linear weighted sum of their inputs, the output is non-linear. If the value is less than 0 the output is 0, if it is greater than 0, the output is the weighted sum.

The first half of the graph is flat, the second is a positive sloped line.

Sigmoid Neurons

Give a real-valued output that is a smooth and bounded function of their total input. Typically they use the logistic function.

y = 1 / (1 + e^-z)

• z: BTN value

The graph is kind of an S shape. It's never above 1 or below 0.

Stochastic Binary Neurons

Use a SN to calculate a probability value then "dice rolling" to see if the output is 1 or 0.

1D: A simple example of learning

Simple network for recognizing numbers. For training; we increase weights for the thing that is right and we decrement weights for the thing that it guessed.

1E: Three types of learning

• Supervised learning
  • Trying to predict an output when given an input vector
• Unsupervised learning
  • Select action to maximize payoff
• Reinforcement learning
  • Discover a good internal representation of the input

Supervised learning has regression and classification. Regression means the output is a real value.

In reinforcement learning the only supervisory signal is an occasional scalar reward. The goal in selecting each action is to maximize the expected sum of future rewards (discounted). this id difficult. Hard to know where we went wrong. A scalar reward does not supply much information. It is not covered in this course.

Unsupervised learning provides a compact low-dimensional representation of the input. Good for finding sensible clusters of the input.

2A: Network Architecture

• Feed forward
• Recurrent
• Symmetrically connected networks

Fee forward - has input and output units with one or more hidden layers of neurons. If there is more than one hidden layer, it is a "deep" neural network.

Recurrent networks can be hard to train, but they can be very powerful. Recurrent networks are a good way to model time.

These have the same connection going each way and are easier to analize.

2B: Perceptrons

Network with a single hidden layer. Features - the connections from the input to the hidden layer and hand picked, and the connections from the hidden layer to the output is learned.

Rosenblatt made extreme claims in the early 60's. Minsky and Papert showed that they could not in 69 but people thought it applied to all neural networks (not just perceptrons). Sill widely used.

The decision unit is a Binary Threshold unit. We use an additional weight to a neuron that is always on as the bias.

The perceptrong convergence procedure.

1. Add an extra component with value 1 to each input to serve as the bias.
2. Pick training cases that so that every case gets hit
   1. If output is correct, leave it alone
   2. If the output unit incorrectly outputs a zero, add teh input vector to the weight vector (including the bias weight)
   3. If the output unit incorrectly outputs a 1, subtract the input vector from the weight vector (including the bias weight)

The guarantees to find a correct set of weights if such a set exists.

2C: Geometrical View of Perceptrons

Weights can be though of planes, they cut off spaces that cannot find the correct answer. If they cut off all space, there is no answer. If they produce a cone there is a solution. If you have two solutions, then the average of those two is also a solution - it is a convex problem.

2D: Why the learning works

Think of how wrong a solution is by how far the current solution is from every feasible weight vectors. This doesn't work because some values will move further away from some values. So we add a generous margin so that won't happen. I don't care about this lecture.

2E: What perceptrons can't do

The biggest limitation is that you have to hand choose the features. You can do almost anything if you make every possible feature unit, but that makes the problem exponentially large.

It cannot identify NXOR (or XOR, I think). Because you can't place a plane to select the two points.

They cannot handle a "wrap-around". This is devastating because the whole point of pattern recognition is to learn patterns despite translation.

So perceptrons are cool, but what we need is a way to learn feature detectors.

3a : Learning weights of a linear neuron

In short, perceptron techniques do not work at all here.

Lunch example (finding prices of food based on the cost of a meal over several days).

Delta rule - choose a fraction and move towards the right answer based on the input. GO BACK AND FILL THIS IN

3b : Error Surface for a linear neuron

The error surface is a quadratic bowl. We move along it to try and minimize error.

3c : Learning the weights of a logistic output neuron

The logistic neuron is an S curve going from 0 to 1, equal to .5 at 0.

3d : The backpropagation algorithm

Similar as forward propagation, but reversed. Important slide. Use error derivatives with respect to hidden activities. I'm a little confused on how the derivatives are being computed, but it seems like a lot of the calculation cancels it self out and the final value is y_i * (dE / dZ_j) where E is error, y_i is the activation value of a specific input to J (the output node of our calculation) and Z_j is the total input to J.

3e : Using backpropagation

Mini-batch, use small random samples of training data early and use more later.

Sampling error - there will be accidental regularities just because of the particular training cases.

4a : Family Tree

Shows that back propagation can learn features.

4b : Cognitive Science

Discussions of what ANNs show us about BNN.

4c : Softmax output function

A group of neurons that converts values to percentages. Cross-entropy cost function. REVIEW THIS SHIT

4d : Neuro-probabilistic

More softmax and architecture.

4e : More NP

More on features. Word features. Markov like stuff. It would be cool to see this combined with Markov.

5a : Why object recognition is difficult

• Segmentation : real scenes are cluttered, it's hard to tell where an objects begin and end and parts may be hidden
• Lighting : intensities may vary independently of the object
• Deformation: Objects can take on a lot of forms
• Affordance: objects are often defined by how they are used rather than how they look
• Viewpoint: Changes in viewpoint cause changes in images

5b : Ways to achieve viewpoint invariance

Select features. Put a box around the object (with translation, rotation, scale, shear) but in order to put a box around it we need to recognize the object, which doesn't help with object recognition.

5c : Convolutional neural networks

Based on replicated feature detectors. We have a lot of copies of the same feature detector in many positions in the hidden layer. We must modify our gradient computation to hold the weights equal. It's easy, however, to do this.

Se Le Net (http://yann.lecun.com LENET).

Pooling the output of several replications of a single feature has some uses. But we lose positional information - ie. If we have a "line" feature detector, pooling will allow us to know that the image has a line, but by pooling, we won't know where the line is.

It's important to consider not just how many errors a net makes, but which errors it makes.

A training technique is to use a simulator to generate synthetic data.

5d : Convolution networks for object recognition

Talking about state-of-the-art in object recognition.

Dropout: During training, remove half the hidden units each time so that the remaining units don't learn to compensate.

6a : Gradient Descent

for linear neuron with squared error, error surfaces are a quadratic bowl. For multi-layer, non-linear nets, the error surface is complicated, but locally it will still be a quadratic bowl.

Stochastic gradient descent. Mini-batches are more effective, particularly on a GPU. Avoid "clustered mini-batches". Learning rates should be adjusted down if error rates are jumping wildly or oscillating, and up if they are too consistent and small.

6b : Bag of tricks for mini-batch

If two hidden units have the same bias, incoming and outgoing weights, they will always get the same gradient, so we initialize them with small random values - generally the sqrt of ?fan-in?.

For steepest descent, shifting the input values can change the shape of the error surface from an elongated elipse to more circular. We can add constants or scale the input.

Momentum: Instead of using steepest descent, to change position, we use it change velocity.

We can use different learning rates for different parameters.

THIS WAS GOOD - REWATCH

6c : Momentum method

Add gradient descent to velocity and to add viscosity to slow it down over time, hopefully stopping at the local minima. A small learning rate with large momentum can be very effective.

A better method reverses the order, moving first, then calculating gradient.

6d : Separate learning rates

These are good.

6e : rmsprop

rprop: Using only the sign of the gradient and use a step size adapted to each weight. We increase the step size if the sign of the last two gradients agree, and decrease if they disagree. Place hard bounds on the limits. This doesn't work well with mini-batches though.

rmsprop: Keep a moving average of each weight.

Learning Methods

• Full Batch : small database or large, non-redundant data sets
• Mini-Batch : big, redundant sets using rmsprop

There are no clear learning algorithms.

7a : Modeling sequences

Temporal phenomena sequences seem natural, they also work for other things.

Recurrent networks can have memory.

Neural networks can be better than hidden Markov models because an HMM needs a lot of states to deal with large quantities of data.

The computational power of RNN makes them very hard to train.

7b : Training RNNs with backpropagation

We can think of an RNN as a layered NN with constrained weights (w1 in each layer is the same). Back propagation is good at this.

But now we have to deal with the issue of initial states.

For training data, at the end, we back-propagate through time until the initial state to get the gradient of error.

7c : Toy Example

Training an RNN to do binary addition

7d : Difficulties training RNN

As you move through the iterations back-propagating, you will compound your over or under correction.

How to handle this

• long short term memory
• Hessian free optimization
• Echo state networks
• Good initialization with momentum

7e : Long short term memory

Use the neural net to control a logistic neuron to a "write gate" to a memory cell. This also applies to a "read gate".

Because we are only remembering on thing per memory cell, we can train on whether that value should have been bigger or smaller.

WTF

• I still don't get the back-propigation algorithm
• I don't know what he means when he says "cost" function - I think cost just means error. Why that is, I don't know. Professional academics are confusing.