Neural Networks Notes

Lectures

1A

Machine learning is good for complicated processes combining weak rules to hit a moving target. The machine learning approach is to mine data for rules and it should work for new cases, not just the cases it was trained on.

Best for learning algorithms:

• Recognizing patterns
  • objects in real scenes
  • facial identities or expressions
  • spoken words
• Anomalies
  • unusual sequences of credit card transactions
  • unusual patterns of sensor readings in nuclear power plants
• Prediction
  • future stock prices
  • which movies a person will like

The MNIST database of hand-written digits is the machine learning equivalent of fruit flies to genetics. TIMIT is the same benchmark for speech recognition.

1B : Real Neurons

A cortical neuron has one axon that branches and dendritic tree that collects input form other neurons. A spike of activity in the axon causes charge to be injected into the post-synaptic neuron. Shen a spike of activity travels along an axon and arrives at a synapse it causes vesicles of transmitter chemicals to be released. There are several kinds of transmitter.

• Each neuron receives input from other neurons (a few neurons are also connect to receptors)
• The effect of each input line on the neuron is controlled by a synaptic weight.
• The synaptic weight adapts so that the whole network learns to perform useful computations.
• You have 10^11 neurons each with about 10^4 weights.

The cortex is modular. Different bits of the cortex do different things. Local damage to the brain has specific effects. But the cortex looks pretty much the same all over. This suggests that there is a uniform learning pattern at work.

1C : Simple models of neurons

To model things we have to idealize them.

Linear Neuron

Simple but misleading.

y = b + Σ x_iw_i

• b: bias
• w: weight
• x: input

The graph is a straight line

Binary Threshold Neurons

First compute a weighted sum of the inputs. If the weighted sum exceeds a threshold, send out a spike.

Graph is a cliff.

Rectified Linear Neurons

Sometimes called linear threshold neurons. Compute a linear weighted sum of their inputs, the output is non-linear. If the value is less than 0 the output is 0, if it is greater than 0, the output is the weighted sum.

The first half of the graph is flat, the second is a positive sloped line.

Sigmoid Neurons

Give a real-valued output that is a smooth and bounded function of their total input. Typically they use the logistic function.

y = 1 / (1 + e^-z)

• z: BTN value

The graph is kind of an S shape. It's never above 1 or below 0.

Stochastic Binary Neurons

Use a SN to calculate a probability value then "dice rolling" to see if the output is 1 or 0.

1D: A simple example of learning

Simple network for recognizing numbers. For training; we increase weights for the thing that is right and we decrement weights for the thing that it guessed.

1E: Three types of learning

• Supervised learning
  • Trying to predict an output when given an input vector
• Unsupervised learning
  • Select action to maximize payoff
• Reinforcement learning
  • Discover a good internal representation of the input

Supervised learning has regression and classification. Regression means the output is a real value.

In reinforcement learning the only supervisory signal is an occasional scalar reward. The goal in selecting each action is to maximize the expected sum of future rewards (discounted). this id difficult. Hard to know where we went wrong. A scalar reward does not supply much information. It is not covered in this course.

Unsupervised learning provides a compact low-dimensional representation of the input. Good for finding sensible clusters of the input.

2A: Network Architecture

• Feed forward
• Recurrent
• Symmetrically connected networks

Fee forward - has input and output units with one or more hidden layers of neurons. If there is more than one hidden layer, it is a "deep" neural network.

Recurrent networks can be hard to train, but they can be very powerful. Recurrent networks are a good way to model time.

These have the same connection going each way and are easier to analize.

2B: Perceptrons

Network with a single hidden layer. Features - the connections from the input to the hidden layer and hand picked, and the connections from the hidden layer to the output is learned.

Rosenblatt made extreme claims in the early 60's. Minsky and Papert showed that they could not in 69 but people thought it applied to all neural networks (not just perceptrons). Sill widely used.

The decision unit is a Binary Threshold unit. We use an additional weight to a neuron that is always on as the bias.

The perceptrong convergence procedure.

1. Add an extra component with value 1 to each input to serve as the bias.
2. Pick training cases that so that every case gets hit
   1. If output is correct, leave it alone
   2. If the output unit incorrectly outputs a zero, add teh input vector to the weight vector (including the bias weight)
   3. If the output unit incorrectly outputs a 1, subtract the input vector from the weight vector (including the bias weight)

The guarantees to find a correct set of weights if such a set exists.

2C: Geometrical View of Perceptrons

Weights can be though of planes, they cut off spaces that cannot find the correct answer. If they cut off all space, there is no answer. If they produce a cone there is a solution. If you have two solutions, then the average of those two is also a solution - it is a convex problem.

2D: Why the learning works

Think of how wrong a solution is by how far the current solution is from every feasible weight vectors. This doesn't work because some values will move further away from some values. So we add a generous margin so that won't happen. I don't care about this lecture.

2E: What perceptrons can't do

The biggest limitation is that you have to hand choose the features. You can do almost anything if you make every possible feature unit, but that makes the problem exponentially large.

It cannot identify NXOR (or XOR, I think). Because you can't place a plane to select the two points.

They cannot handle a "wrap-around". This is devastating because the whole point of pattern recognition is to learn patterns despite translation.

So perceptrons are cool, but what we need is a way to learn feature detectors.

3a : Learning weights of a linear neuron

In short, perceptron techniques do not work at all here.

Lunch example (finding prices of food based on the cost of a meal over several days).

Delta rule - choose a fraction and move towards the right answer based on the input. GO BACK AND FILL THIS IN

3b : Error Surface for a linear neuron

The error surface is a quadratic bowl. We move along it to try and minimize error.