ADALINE
ADALINE (Adaptive Linear Neuron or later Adaptive Linear Element) is an early single-layer artificial neural network and the name of the physical device that implemented this network. The network uses memistors. It was developed by Professor Bernard Widrow and his doctorate student Ted Hoff at Stanford University in 1960. It is based on the McCulloch–Pitts neuron. It consists of a weight, a bias and a summation function. The difference between Adaline and the standard (McCulloch–Pitts) perceptron is that in the learning phase, the weights are adjusted according to the weighted sum of the inputs (the net ). In the standard perceptron, the net is passed to the activation (transfer) function and the function's output is used for adjusting the weights. A multilayer network of ADALINE units is known as a MADALINE. Definition. Adaline is a single layer neural network with multiple nodes where each node accepts multiple inputs and generates one output. Given the following variables as: then we find that the output is formula_6. If we further assume that then the output further reduces to: formula_9 Learning algorithm. Let us assume: then the weights are updated as follows formula_13. The ADALINE converges to the least  squares error which is formula_14. This update rule is in fact the stochastic gradient descent update for linear regression. MADALINE. MADALINE (Many ADALINE) is a three-layer (input, hidden, output), fully connected, feed-forward artificial neural network architecture for classification that uses ADALINE units in its hidden and output layers, i.e. its activation function is the sign function. The three-layer network uses memistors. Three different training algorithms for MADALINE networks, which cannot be learned using backpropagation because the sign function is not differentiable, have been suggested, called Rule I, Rule II and Rule III. MADALINE Rule 1 (MRI ) - The first of these dates back to 1962 and cannot adapt the weights of the hidden-output connection. MADALINE Rule 2 (MRII) - The second training algorithm improved on Rule I and was described in 1988. The Rule II training algorithm is based on a principle called "minimal disturbance". It proceeds by looping over training examples, then for each example, it: MADALINE Rule 3 - The third "Rule" applied to a modified network with sigmoid activations instead of signum; it was later found to be equivalent to backpropagation. Additionally, when flipping single units' signs does not drive the error to zero  for a particular example, the training algorithm starts flipping pairs of units' signs, then triples of units, etc.