Neural Networks Notes - Revision history

Adamcolton: /* WTF */

2013-05-01T15:07:59Z

WTF

2013-05-01T15:07:08Z

WTF

2013-02-15T20:19:19Z

2012-10-29T20:44:05Z

← Older revision		Revision as of 20:44, 29 October 2012
Line 232:		Line 232:

	For training data, at the end, we back-propagate through time until the initial state to get the gradient of error.		For training data, at the end, we back-propagate through time until the initial state to get the gradient of error.

			===7c : Toy Example===
			Training an RNN to do binary addition

			===7d : Difficulties training RNN===
			As you move through the iterations back-propagating, you will compound your over or under correction.

			How to handle this
			*long short term memory
			*Hessian free optimization
			*Echo state networks
			*Good initialization with momentum

			===7e : Long short term memory===
			Use the neural net to control a logistic neuron to a "write gate" to a memory cell. This also applies to a "read gate".

			Because we are only remembering on thing per memory cell, we can train on whether that value should have been bigger or smaller.


	==WTF==		==WTF==

2012-10-29T20:08:24Z

← Older revision		Revision as of 20:08, 29 October 2012
Line 216:		Line 216:

	There are no clear learning algorithms.		There are no clear learning algorithms.

			===7a : Modeling sequences===
			Temporal phenomena sequences seem natural, they also work for other things.

			Recurrent networks can have memory.

			Neural networks can be better than hidden Markov models because an HMM needs a lot of states to deal with large quantities of data.

			The computational power of RNN makes them very hard to train.

			===7b : Training RNNs with backpropagation===
			We can think of an RNN as a layered NN with constrained weights (w1 in each layer is the same). Back propagation is good at this.

			But now we have to deal with the issue of initial states.

			For training data, at the end, we back-propagate through time until the initial state to get the gradient of error.

	==WTF==		==WTF==
	*I still don't get the back-propigation algorithm		*I still don't get the back-propigation algorithm
	*I don't know what he means when he says "cost" function		*I don't know what he means when he says "cost" function - I think cost just means error. Why that is, I don't know. Professional academics are confusing.

	[[Category:Notes]]		[[Category:Notes]]
	[[Category:Class]]		[[Category:Class]]

2012-10-29T18:08:35Z

← Older revision		Revision as of 18:08, 29 October 2012
Line 216:		Line 216:

	There are no clear learning algorithms.		There are no clear learning algorithms.

			==WTF==
			*I still don't get the back-propigation algorithm
			*I don't know what he means when he says "cost" function

	[[Category:Notes]]		[[Category:Notes]]
	[[Category:Class]]		[[Category:Class]]

2012-10-29T17:36:33Z

5c : Convolutional neural networks

← Older revision		Revision as of 17:36, 29 October 2012
Line 167:		Line 167:

	===5c : Convolutional neural networks===		===5c : Convolutional neural networks===
	Based on replicated feature detectors. We must modify our gradient computation to hold the weights equal.		Based on replicated feature detectors. We have a lot of copies of the same feature detector in many positions in the hidden layer. We must modify our gradient computation to hold the weights equal. It's easy, however, to do this.

	Se Le Net (http://yann.lecun.com LENET).		Se Le Net (http://yann.lecun.com LENET).

	Pooling.		Pooling the output of several replications of a single feature has some uses. But we lose positional information - ie. If we have a "line" feature detector, pooling will allow us to know that the image has a line, but by pooling, we won't know where the line is.

	It's important to consider not just how many errors a net makes, but which errors it makes.		It's important to consider not just how many errors a net makes, but which errors it makes.

			A training technique is to use a simulator to generate synthetic data.

	===5d : Convolution networks for object recognition===		===5d : Convolution networks for object recognition===

2012-10-26T01:34:21Z

4c : Softmax output function

← Older revision		Revision as of 01:34, 26 October 2012
Line 148:		Line 148:

	===4c : Softmax output function===		===4c : Softmax output function===
	A group of neurons that converts values to percentages. Cross-entropy cost function.		A group of neurons that converts values to percentages. Cross-entropy cost function. REVIEW THIS SHIT

	===4d : Neuro-probabilistic===		===4d : Neuro-probabilistic===

2012-10-18T16:13:52Z

@@ Line 181: / Line 181: @@
 ===6a : Gradient Descent===
 [[Category:Notes]]
 [[Category:Class]]

2012-10-18T15:04:05Z

← Older revision		Revision as of 15:04, 18 October 2012
Line 176:		Line 176:

	===5d : Convolution networks for object recognition===		===5d : Convolution networks for object recognition===
			Talking about state-of-the-art in object recognition.

			Dropout: During training, remove half the hidden units each time so that the remaining units don't learn to compensate.

			===6a : Gradient Descent===

	[[Category:Notes]]		[[Category:Notes]]
	[[Category:Class]]		[[Category:Class]]